Date   

End of Google Summer of Code - 2020

Shaheem Azmal M MD
 

Hello Everyone,

 

We would like to congratulate Darshan, Kaushlendra and Ayush for their contributions to FOSSology community. We are very impressed by the overall quality of their work and organizational skills, keep up the good work and stay in touch with community.

 

Details of features that were developed in this GSOC-2020.

 

Project : FOSSology

Student Name : Darshan Kansagara

Feature Title : Dashboard (Python, influxDB, Grafana)

Wiki : https://github.com/darshank15/GSoC_2020_FOSSOlogy/wiki

 

Project : Atarashi

Student Name : Ayush (HastagAB)

Feature Title : Accelerating Atarashi

Wiki : https://github.com/hastagAB/GSoC-2020

 

Project : Atarashi

Student Name : Kaushlendra Pratap Singh

Feature Title : Refurbishing Atarashi

Wiki : https://github.com/Kaushl2208/GSoC-2020

 

We would also like to thank mentors(Michael, Shaheem, Anupam, Gaurav, Nicolas, Aman and Sandeep) for helping students achieve their targets.

 

 

Best Regards

Shaheem Azmal M MD

 

 

 


Re: Not able to upload file >700MB file even after changing values in php.ini file

Michael C. Jaeger
 

hello,

ja a quick question: if the uploa dis 1.7Gb and the max mem limit is 1.0 GB then this would be the problem. Pls. try setting the mem limit only few GBs above post max size. (something like 2.2 GB, 2.1 GB and and 2.0 GB.

And you re right postgresql needs different setting for uploads of 1.7Gb size.

I am not sure about work meme of 2GB 8could be also Ok with 256MB), maybe you re not going to use 6GB of maintenance mem.

but a good approach is use a tool at first to get you reaosnable settings for the machine youre using:

https://pgtune.leopard.in.ua/#/

was a good help for me in the past for tuning postgresql. Please note that you re using this link at your own risk.

Kind regards, Michael

On 21. Jul 2020, at 01:12, Praba <@dpraba> wrote:

Hello,

Just started to work on Fossology tool last few days.
I am trying to upload linux5.4.30 component which is around 1.7G and not able to upload the file even after changing “post_max_size” & “upload_max_filesize”.
I was able to upload and scan other components less than 700MB.

I have Fossology installed using docker on R640 Dell server.
FOSSology version 3.8.0-86-gf0c84e229 (code revision f0c84e)

Below are the details. Please provide details if any modifications are needed.

root@abbf7f77a767:/fossology# cat /etc/php/7.0/apache2/php.ini|grep memory_limit
memory_limit = 1002M

root@abbf7f77a767:/fossology# cat /etc/php/7.0/apache2/php.ini|grep post_max_size
post_max_size = 4001M

root@abbf7f77a767:/fossology# cat /etc/php/7.0/apache2/php.ini|grep upload_max_filesize
upload_max_filesize = 3000M

Also made some tunable and config changes to few files.
/etc/sysctl.conf
kernel.shmmax=67328311296
kernel.shmall=16437576

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep max_connections
max_connections = 100 # (change requires restart)

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep shared_buffers
#shared_buffers = 128MB # min 128kB
shared_buffers = 32GB # min 128kB
#wal_buffers = -1 # min 32kB, -1 sets based on shared_buffers

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep effective_cache_size
#effective_cache_size = 4GB
effective_cache_size = 10GB

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep work_mem
#work_mem = 4MB # min 64kB
work_mem = 2GB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
maintenance_work_mem = 6GB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep maintenance_work_mem
#maintenance_work_mem = 64MB # min 1MB
maintenance_work_mem = 6GB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep fsync
#fsync = on # flush data to disk for crash safety
fsync = on # flush data to disk for crash safety
#wal_sync_method = fsync # the default is the first option
# fsync
# fsync_writethrough
#data_sync_retry = off # retry or panic on failure to fsync

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep full_page_writes
#full_page_writes = on # recover from partial page writes
full_page_writes = off # recover from partial page writes

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep autovacuum
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
#autovacuum = on # Enable autovacuum subprocess? 'on'
autovacuum = on # Enable autovacuum subprocess? 'on'
#log_autovacuum_min_duration = -1 # -1 disables, 0 logs all actions and
..

Thanks!


Not able to upload file >700MB file even after changing values in php.ini file

Praba
 

Hello,

 

Just started to work on Fossology tool last few days.

I am trying to upload linux5.4.30 component which is around 1.7G and not able to upload the file even after changing “post_max_size” & “upload_max_filesize”.

I was able to upload and scan other components less than 700MB.

 

I have Fossology installed using docker on R640 Dell server.

FOSSology version 3.8.0-86-gf0c84e229 (code revision f0c84e)

 

Below are the details. Please provide details if any modifications are needed.

 

root@abbf7f77a767:/fossology#    cat /etc/php/7.0/apache2/php.ini|grep memory_limit

memory_limit = 1002M

 

root@abbf7f77a767:/fossology# cat /etc/php/7.0/apache2/php.ini|grep post_max_size

post_max_size = 4001M

 

root@abbf7f77a767:/fossology# cat /etc/php/7.0/apache2/php.ini|grep upload_max_filesize

upload_max_filesize = 3000M

 

Also made some tunable and config changes to few files.

/etc/sysctl.conf

kernel.shmmax=67328311296

kernel.shmall=16437576

 

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep max_connections

max_connections = 100                   # (change requires restart)

 

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep shared_buffers

#shared_buffers = 128MB                 # min 128kB

shared_buffers = 32GB                   # min 128kB

#wal_buffers = -1                       # min 32kB, -1 sets based on shared_buffers

 

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep effective_cache_size

#effective_cache_size = 4GB

effective_cache_size = 10GB

 

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep work_mem

#work_mem = 4MB                         # min 64kB

work_mem = 2GB                          # min 64kB

#maintenance_work_mem = 64MB            # min 1MB

maintenance_work_mem = 6GB              # min 1MB

#autovacuum_work_mem = -1               # min 1MB, or -1 to use maintenance_work_mem

 

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep maintenance_work_mem

#maintenance_work_mem = 64MB            # min 1MB

maintenance_work_mem = 6GB              # min 1MB

#autovacuum_work_mem = -1               # min 1MB, or -1 to use maintenance_work_mem

 

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep fsync

#fsync = on                             # flush data to disk for crash safety

fsync = on                              # flush data to disk for crash safety

#wal_sync_method = fsync                # the default is the first option

                                        #   fsync

                                        #   fsync_writethrough

#data_sync_retry = off                  # retry or panic on failure to fsync

 

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep full_page_writes

#full_page_writes = on                  # recover from partial page writes

full_page_writes = off                  # recover from partial page writes

 

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep autovacuum

#autovacuum_work_mem = -1               # min 1MB, or -1 to use maintenance_work_mem

#autovacuum = on                        # Enable autovacuum subprocess?  'on'

autovacuum = on                         # Enable autovacuum subprocess?  'on'

#log_autovacuum_min_duration = -1       # -1 disables, 0 logs all actions and

..

 

Thanks!


MIT License

Ashkar Dev
 

Hi, This question is about OpenSource License
I am on developing a project and used MIT Licensed codes of others and I collected the Licenses but now I'm not sure which code is with which license and also the code was may be modified

is it ok to add all licenses to a place on a web page? instead of making it comment?
in every page footer, make a link to the page of licenses?
and adding the link to that page at the top of CSS files
so it matches the MIT License condition that says substantial portions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

and I want to create a page with licenses and references to the original code
like this:
this project may include these licenses :    
1) --- with reference
2) --- with reference
......


Fossology license manager

surendra bhargava
 

Hi,

I have come across that fossology can be used as a license manager.
How is fossology helpful as a license manager.It is only a compliance tool that scans and reports the detected licenses, is it?

Thanks.
Surendra


fossology license manager

surendra bhargava
 

I have come across that fossology can be used as a license manager.
How is fossology helpful as a license manager.It is only a compliance tool that scans and reports the detected licenses, is it?


FOSSology 3.7.0 release

Shaheem Azmal M MD
 

Hello everyone,

We have released FOSSology 3.7.0 yesterday.

As for the release 3.7.0 of FOSSology, most notable additions since are:
 
  • The scanner that finds dedicated SPDX-License-Ref statements in files is now supported by the auto-decision mode in FOSSology. Thus generated SPDX documents will not only have the information about licenses found by scanners, but also allow for automatically adding conclusions. Note that this can be triggered also by the REST API. 
  • The REST API has been adapted to better integrate with SW360. This version of FOSSology is suitable for use with SW360 6.0 in order to have a working integration between the two.
 

Contributors

 
Credits go to the following persons for this release since 3.7.0-RC1:
 
Anupam Ghosh <anupam.ghosh@...>
Gaurav Mishra <mishra.gaurav@...>
Martin Michlmayr <tbm@...>
Maximilian Huber <maximilian.huber@...>
Michael C. Jaeger <michael.c.jaeger@...>
Shaheem Azmal M MD <shaheem.azmal@...>

Please find the release and binary packages for Debian and Ubuntu based systems on release page.


Regards,
FOSSology Team


Fossology REST Clients for .NET

Graf, Thomas
 

Dear all,

 

I like to inform you, that I have added a .NET implementation of the FOSSology REST API

to the FOSSology GitHub project:

https://github.com/fossology/FOSSology.REST.dotnet

 

The implementation covers most of the REST commands.

The unit tests will run on FOSSology in a docker container. Our goal is to add one further step

to the automation of software license compliance.

 

Do not hesitate to drop me an email in case of questions or suggestions.

 

Best regards,


Thomas


Siemens AG
Smart Infrastructure
mailto:thomas.graf@...


Google Summer of Code final report, Integration of Software Heritage in FOSSology

Sandip Bhuyan
 

Hello Everyone,
I am Sandip Kumar Bhuyan was a Google Summer of Code 2019 intern for fossology organization. Thank you for selecting my proposal for this year GSoC. I think I have reached the expectation. I was working on integrating software heritage in fossology. I have completed the GSoC 2019 successfully. You can find my report in my blogpost Blog Post. I hope everyone will like it.
Cheers
Sandip

--
Sandip Kumar Bhuyan | sandipbhuyan@... | sandipbhuyan.com
Have a great day| Code better


FOSSology 3.6.0 release

Shaheem Azmal M MD
 

Hello everyone,

After two release candidates, making fixes for migration tests, unified report and load issues with tree-view, FOSSology is stable enough for a new release. The main features of the 3.6.0 release can be found under

[RC1](https://github.com/fossology/fossology/releases/tag/3.6.0-rc1). Particular corrections after RC1 can be found under [RC2](https://github.com/fossology/fossology/releases/tag/3.6.0-rc2).

 

Few interesting features in this release are: 

 

  • A new agent named `ojo` (eye in Spanish) which does dedicated searches for the 'SPDX-License-Identifier' statements
  • Improved handling of manually added copyright statements to files
  • Improvements to the SPDX reporting, for example output also of comments
  • Calculating the SHA256 values for files from now on, because that is going to be used for integration of, for example, Software Heritage or Clearly defined

 

Credits to 3.6.0

 

  From the git commit history, we have following contributors since 3.5.0:

 

  > @andi8086 <andreas.reichel@...>,

  > @ag4ums <anupam.ghosh@...>,

  > @hastagAB <classicayush@...>,

  > @chienphamvu <chienphamvu@...>,

  > @ChristopheRequillart <christophe.requillart@...>,

  > @GMishx <mishra.gaurav@...>,

  > @maxhbr <maximilian.huber@...>,

  > @mcjaeger <michael.c.jaeger@...>,

  > @NicolasToussaint <nicolas1.toussaint@...>,

  > @PeterDaveHello <hsu@...>,

  > @rlintu <raino.lintulampi@...>,

  > @sandipbhuyan <sandipbhuyan@...>,

  > @shaheemazmalmmd <shaheem.azmal@...>,

  > @tiegz <tieg@...>,

  > @vivekaindia <vvksindia@...>

Please find the release and binary packages for Debian and Ubuntu based systems [here](https://github.com/fossology/fossology/releases/tag/3.6.0).


Regards,
Shaheem Azmal M MD

 
 


GSoC 19 - Spasht Agent Documentation

vivek kumar
 


Need to remove Debian packaging meta info from master branch

Gaurav Mishra
 

Hello all,

 

During our effort to publish FOSSology as a Debian package, we got few suggestions from the Debian community.

One of those suggestion is to remove the Debian packaging information (debian folder) from the master branch and put it into another branch like chore/debian/jessie.

 

This is done so to avoid conflicts as Debian maintainers will be editing this packaging information in the FOSSology mirror (hosted at Debian Sala). And any change in upstream can result in conflicts.

 

As this change will alter the packaging steps required by many of FOSSology users, we need your feedback.

 

I have opened an issue on GitHub for the same: https://github.com/fossology/fossology/issues/1341

 

Kindly respond either on this thread or on the GitHub issue if you have any concerns regarding the same.

With best regards,
Gaurav Mishra


Release of 3.5.0

Shaheem Azmal M MD
 

Hello all,

After two release candidates, making fixes for REST API installation and various migration tests, FOSSology is stable enough for a new release. The main features of the 3.5.0 release can be found under RC1.

Particular corrections after RC1 can be found under RC2.

Mainly 3.5.0 adds more documentation, infrastructure improvements and support for brand new FOSSology REST API. A brief introduction about the REST API can be found at:

https://www.fossology.org/get-started/basic-rest-api-calls/

Moreover, new functionality has improved JSON output for nomos and restructured license detection for nomos. Last but not the least, FOSSology now have capabilities to ignore files specific to version control systems from the scanning improving scan times.

Credits

From the git commit history, we have following contributors since 3.4.0:

@ag4ums,
@ChristopheRequillart,
@AMDmi3,
@GMishx,
@mcieno,
@max-wittig,
@maxhbr,
@rlintu,
@sandipbhuyan,
@shaheemazmalmmd


Please find the release and binary packages for Debian and Ubuntu based systems here https://github.com/fossology/fossology/releases/tag/3.5.0 

Thanks & Regards
Shaheem Azmal M MD


Re: Is there any plans to use ALL features of SPDX 2.1 specification in future releases?

Michael C. Jaeger
 

Hello,

currently, I am not aware of the FOSSology project planning to support "ALL features of SPDX 2.1 specification in future releases", for a number of reasons, just a few examples:

* Spec 2.1 supports identification of code snippets (See section 5), currently fossology does not support it and I am not aware of plans by someone to contribute it
* Spec 2.1 support for example besides the copyright statement also a file contributor, which could be taken, maybe, from some SCM information. I am not aware of plans here neither
* ...

and so forth. I think FOSSology will support only parts of the SPDX 2.1 spec as they are covered by the application functionality.

I am not sure of that answer covers your question? The fact that you have used bold types letters in your e-mail provides a slight impression that you expect something in particular from the FOSSology project?

Maybe the following two issues provide also helpful information to you?

https://github.com/fossology/fossology/issues/1309
https://github.com/spdx/spdx-spec/issues/112

Please do not hesitate to clarify what you intended to say about our release notes or our idea of SPDX 2.1 document generation.

Kind regards, Michael

On 12. Mar 2019, at 13:30, thuy.tran.xh@... wrote:

Hi all,

As announced in the release, https://github.com/fossology/fossology/wiki/FOSSology-3.1-Release-Announcement-(Working-Version)

Key features in FOSSology 3.1 are:
Support for SPDX 2.1 document formats(tag:value format now available as well as RDF)

I have tried the fossology installation in the ways of docker as well as from source.
From the template (Only package/ document/file information) at https://github.com/fossology/fossology/tree/master/src/spdx2/agent/template, we could not generate SPDX 2.1 FULL features at https://spdx.org/spdx-specification-21-web-version

Is there any plans to use ALL features of SPDX 2.1 specification in future releases?
Thank you.

Regards,
Thuy Tran.


Is there any plans to use ALL features of SPDX 2.1 specification in future releases?

thuy.tran.xh@...
 

Hi all,

As announced in the release, https://github.com/fossology/fossology/wiki/FOSSology-3.1-Release-Announcement-(Working-Version)

Key features in FOSSology 3.1 are:
Support for SPDX 2.1 document formats(tag:value format now available as well as RDF)

I have tried the fossology installation in the ways of docker as well as from source.
From the template (Only package/ document/file information) at https://github.com/fossology/fossology/tree/master/src/spdx2/agent/template, we could not generate SPDX 2.1 FULL features at https://spdx.org/spdx-specification-21-web-version

Is there any plans to use ALL features of SPDX 2.1 specification in future releases?
Thank you.

Regards,
Thuy Tran.


Re: VS: [FOSSology] Help a newbie

marc.mcgarry@...
 

On Fri, Oct 12, 2018 at 08:05 AM, Martin von Willebrand wrote:

ng will unpack zip-packages and jar-packages, even if they are nested.

 

You likely want to run source code files through Fossology, though, because many build systems remove texts that would be interesting for license analysis purposes. In java, you should look at uploading .java-files and not .class-files.

 

Best


VS: [FOSSology] Help a newbie

Martin von Willebrand
 

Fossology agent doing the unpacking will unpack zip-packages and jar-packages, even if they are nested.

 

You likely want to run source code files through Fossology, though, because many build systems remove texts that would be interesting for license analysis purposes. In java, you should look at uploading .java-files and not .class-files.

 

Best

Martin

 

Martin von Willebrand, Attorney-at-law, Partner
HH Partners, Attorneys-at-law Ltd
Bulevardi 7, 5th floor
P.O. Box 232, 0
0101 Helsinki, Finland
Tel: +358 9 177 613, Fax: +358 9 653 873
GSM: +358 40 770 1818
martin.vonwillebrand@...
www.twitter.com/mvonwi
www.hhpartners.fi
Validos ry, Chairman,
www.validos.org

HH Partners shines in international rankings. See details at hhpartners.fi.


Privileged and confidential information may be contained in this message. If you are not addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, kindly notify us by reply e-mail and delete this message immediately. Thank you.

 

Lähettäjä: main@... [mailto:main@...] Puolesta marc.mcgarry@...
Lähetetty: perjantai 12. lokakuuta 2018 17.57
Vastaanottaja: main@...
Aihe: [FOSSology] Help a newbie

 

I am trying to run software packages/components through fossology. If i zip the package folder (with the jar file inside), will this successfully test all of the classes inside? 

 


Re: Help a newbie

Michael C. Jaeger
 

Hello,

 

oh interesting, I have not seen your point from your answer, glad that our e-mails crossed.

 

Kind regards, Michael

 

From: main@... [mailto:main@...] On Behalf Of Nicolas Toussaint
Sent: Freitag, 12. Oktober 2018 17:03
To: main@...
Subject: Re: [FOSSology] Help a newbie

 

Hi Marc,

Yes, Fossology will open all compressed files (zip, tgz, jar, etc.) and scan the contents.
This is done recursively, so that a jar in a zip is also opened and its contents scanned.


On 12/10/2018 16:56, marc.mcgarry@... wrote:

I am trying to run software packages/components through fossology. If i zip the package folder (with the jar file inside), will this successfully test all of the classes inside? 

 



-- 
 
Nicolas Toussaint
OAB - Orange Applications for Business - Lyon
_________________________________________________________________________________________________________________________
 
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
 
This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.


Re: Help a newbie

Michael C. Jaeger
 

Hello,

 

your mail raises a few questions, I try to bring a few considerations.

 

1. „classes“ refers to *.class files? if yes, then this is maybe not the right thing for FOSSology, but maybe for some binary analysis tool (- next generation) or so. class files are usually results of a compiler so they omit source code comments which is where licensing statements are usually in.

 

2. FOSSology should be able to unpack *.jar files, so no need to zip it, except you would like to upload a set of jar files at once. If fossology cannot look into a jar file, it is a bug (and should be thus put in the issues tracker).

 

3. what is your point with “test” what are look for exactly?

 

Kind regards, Michael

 

From: main@... [mailto:main@...] On Behalf Of marc.mcgarry@...
Sent: Freitag, 12. Oktober 2018 16:57
To: main@...
Subject: [FOSSology] Help a newbie

 

I am trying to run software packages/components through fossology. If i zip the package folder (with the jar file inside), will this successfully test all of the classes inside? 

 


Re: Help a newbie

 

Hi Marc,

Yes, Fossology will open all compressed
files (zip, tgz, jar, etc.) and scan the contents.
This is done recursively, so that a jar in a zip is also opened and its contents scanned.



On 12/10/2018 16:56, marc.mcgarry@... wrote:
I am trying to run software packages/components through fossology. If i zip the package folder (with the jar file inside), will this successfully test all of the classes inside? 

 

-- 

Nicolas Toussaint
OAB - Orange Applications for Business - Lyon
_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.