Date   
Fossology, SPDX Packages & Sub Packages

 

Dear All,

I am having a hard using Fossology to fulfil my needs efficiently.
I would like to share one of the core aspect I am struggling with, and would love to see if some of you are facing the same problems.

My goal is to scan complete products source code, and ultimately produce a report listing all embedded components (libraries, dependencies, etc.), their licenses as well as copyright notices.
Today, I generate SPDX-TV reports with Fossolgy, convert them to XLS format using the SPDX Tools [1], and manually reorganise the file to create a list of components.
I didn't find within Fossology  a way to indicate that, for example, a given directory/file contains the library X, licensed under Y, and copyrighted to Z, and generate the corresponding SPDX report.

Beside, the SPDX specifications seem to allow Packages and Sub-Packages identification [2] that seems to be what I'm looking for.

So my questions are:
- Do you share the need of identifying components / sub-packages within a scanned project ?
- If so, is there a way to achieve this with Fossology, and producing SPDX reports ?
- Can the SPDX [Sub-]Package be used to identify components and their license ?
- ... or is my compliance process totally wrong (should I identify and scan all components separately) ?



Nicolas

-- 
Nicolas Toussaint
OAB - Orange Applications for Business - Lyon
Tel: +33 608 763 559
_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

Re: Fossology, SPDX Packages & Sub Packages

 

Hello again,

I came to the conclusion that, 
- from an SPDX point of view, the packages is what I need to describe the embedded components and dependencies included in the scanned project
- Fossology does not permit (yet) the creation of multiple packages

Now we can head towards making Fossology handle such packages + exoprt them to SPDX, but I will need confirmation that it is a good idea & others have the same need.

Also, how to achieve this ? 
I can imagine adding a "Make Package" link in the Actions column (see attached file).
This would 
- let the user configure the package
- clear all files within the directory (or compressed file), and include them in the package

In an ideal world (to come), dependencies and library could be compared to existing database (see Sharing-creates-value and Clearly Defined initiatives) and cleared automatically this way.

Any thoughts on this ?





-- 
Nicolas Toussaint
OAB - Orange Applications for Business - Lyon

-----Original Message-----
From: Nicolas Toussaint <nicolas1.toussaint@...>
Subject: Fossology, SPDX Packages & Sub Packages
Date: Mon, 20 Aug 2018 16:01:28 +0200

Dear All,

I am having a hard using Fossology to fulfil my needs efficiently.
I would like to share one of the core aspect I am struggling with, and would love to see if some of you are facing the same problems.

My goal is to scan complete products source code, and ultimately produce a report listing all embedded components (libraries, dependencies, etc.), their licenses as well as copyright notices.
Today, I generate SPDX-TV reports with Fossolgy, convert them to XLS format using the SPDX Tools [1], and manually reorganise the file to create a list of components.
I didn't find within Fossology  a way to indicate that, for example, a given directory/file contains the library X, licensed under Y, and copyrighted to Z, and generate the corresponding SPDX report.

Beside, the SPDX specifications seem to allow Packages and Sub-Packages identification [2] that seems to be what I'm looking for.

So my questions are:
- Do you share the need of identifying components / sub-packages within a scanned project ?
- If so, is there a way to achieve this with Fossology, and producing SPDX reports ?
- Can the SPDX [Sub-]Package be used to identify components and their license ?
- ... or is my compliance process totally wrong (should I identify and scan all components separately) ?



Nicolas

-- 
Nicolas Toussaint
OAB - Orange Applications for Business - Lyon
Tel: +33 608 763 559

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

Help a newbie

marc.mcgarry@...
 

I am trying to run software packages/components through fossology. If i zip the package folder (with the jar file inside), will this successfully test all of the classes inside? 

 

Re: Help a newbie

 

Hi Marc,

Yes, Fossology will open all compressed
files (zip, tgz, jar, etc.) and scan the contents.
This is done recursively, so that a jar in a zip is also opened and its contents scanned.



On 12/10/2018 16:56, marc.mcgarry@... wrote:
I am trying to run software packages/components through fossology. If i zip the package folder (with the jar file inside), will this successfully test all of the classes inside? 

 

-- 

Nicolas Toussaint
OAB - Orange Applications for Business - Lyon
_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

Re: Help a newbie

Michael C. Jaeger
 

Hello,

 

your mail raises a few questions, I try to bring a few considerations.

 

1. „classes“ refers to *.class files? if yes, then this is maybe not the right thing for FOSSology, but maybe for some binary analysis tool (- next generation) or so. class files are usually results of a compiler so they omit source code comments which is where licensing statements are usually in.

 

2. FOSSology should be able to unpack *.jar files, so no need to zip it, except you would like to upload a set of jar files at once. If fossology cannot look into a jar file, it is a bug (and should be thus put in the issues tracker).

 

3. what is your point with “test” what are look for exactly?

 

Kind regards, Michael

 

From: main@... [mailto:main@...] On Behalf Of marc.mcgarry@...
Sent: Freitag, 12. Oktober 2018 16:57
To: main@...
Subject: [FOSSology] Help a newbie

 

I am trying to run software packages/components through fossology. If i zip the package folder (with the jar file inside), will this successfully test all of the classes inside? 

 

Re: Help a newbie

Michael C. Jaeger
 

Hello,

 

oh interesting, I have not seen your point from your answer, glad that our e-mails crossed.

 

Kind regards, Michael

 

From: main@... [mailto:main@...] On Behalf Of Nicolas Toussaint
Sent: Freitag, 12. Oktober 2018 17:03
To: main@...
Subject: Re: [FOSSology] Help a newbie

 

Hi Marc,

Yes, Fossology will open all compressed files (zip, tgz, jar, etc.) and scan the contents.
This is done recursively, so that a jar in a zip is also opened and its contents scanned.


On 12/10/2018 16:56, marc.mcgarry@... wrote:

I am trying to run software packages/components through fossology. If i zip the package folder (with the jar file inside), will this successfully test all of the classes inside? 

 



-- 
 
Nicolas Toussaint
OAB - Orange Applications for Business - Lyon
_________________________________________________________________________________________________________________________
 
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
 
This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

VS: [FOSSology] Help a newbie

Martin von Willebrand
 

Fossology agent doing the unpacking will unpack zip-packages and jar-packages, even if they are nested.

 

You likely want to run source code files through Fossology, though, because many build systems remove texts that would be interesting for license analysis purposes. In java, you should look at uploading .java-files and not .class-files.

 

Best

Martin

 

Martin von Willebrand, Attorney-at-law, Partner
HH Partners, Attorneys-at-law Ltd
Bulevardi 7, 5th floor
P.O. Box 232, 0
0101 Helsinki, Finland
Tel: +358 9 177 613, Fax: +358 9 653 873
GSM: +358 40 770 1818
martin.vonwillebrand@...
www.twitter.com/mvonwi
www.hhpartners.fi
Validos ry, Chairman,
www.validos.org

HH Partners shines in international rankings. See details at hhpartners.fi.


Privileged and confidential information may be contained in this message. If you are not addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, kindly notify us by reply e-mail and delete this message immediately. Thank you.

 

Lähettäjä: main@... [mailto:main@...] Puolesta marc.mcgarry@...
Lähetetty: perjantai 12. lokakuuta 2018 17.57
Vastaanottaja: main@...
Aihe: [FOSSology] Help a newbie

 

I am trying to run software packages/components through fossology. If i zip the package folder (with the jar file inside), will this successfully test all of the classes inside? 

 

Re: VS: [FOSSology] Help a newbie

marc.mcgarry@...
 

On Fri, Oct 12, 2018 at 08:05 AM, Martin von Willebrand wrote:

ng will unpack zip-packages and jar-packages, even if they are nested.

 

You likely want to run source code files through Fossology, though, because many build systems remove texts that would be interesting for license analysis purposes. In java, you should look at uploading .java-files and not .class-files.

 

Best

Is there any plans to use ALL features of SPDX 2.1 specification in future releases?

thuy.tran.xh@...
 

Hi all,

As announced in the release, https://github.com/fossology/fossology/wiki/FOSSology-3.1-Release-Announcement-(Working-Version)

Key features in FOSSology 3.1 are:
Support for SPDX 2.1 document formats(tag:value format now available as well as RDF)

I have tried the fossology installation in the ways of docker as well as from source.
From the template (Only package/ document/file information) at https://github.com/fossology/fossology/tree/master/src/spdx2/agent/template, we could not generate SPDX 2.1 FULL features at https://spdx.org/spdx-specification-21-web-version

Is there any plans to use ALL features of SPDX 2.1 specification in future releases?
Thank you.

Regards,
Thuy Tran.

Re: Is there any plans to use ALL features of SPDX 2.1 specification in future releases?

Michael C. Jaeger
 

Hello,

currently, I am not aware of the FOSSology project planning to support "ALL features of SPDX 2.1 specification in future releases", for a number of reasons, just a few examples:

* Spec 2.1 supports identification of code snippets (See section 5), currently fossology does not support it and I am not aware of plans by someone to contribute it
* Spec 2.1 support for example besides the copyright statement also a file contributor, which could be taken, maybe, from some SCM information. I am not aware of plans here neither
* ...

and so forth. I think FOSSology will support only parts of the SPDX 2.1 spec as they are covered by the application functionality.

I am not sure of that answer covers your question? The fact that you have used bold types letters in your e-mail provides a slight impression that you expect something in particular from the FOSSology project?

Maybe the following two issues provide also helpful information to you?

https://github.com/fossology/fossology/issues/1309
https://github.com/spdx/spdx-spec/issues/112

Please do not hesitate to clarify what you intended to say about our release notes or our idea of SPDX 2.1 document generation.

Kind regards, Michael

On 12. Mar 2019, at 13:30, thuy.tran.xh@... wrote:

Hi all,

As announced in the release, https://github.com/fossology/fossology/wiki/FOSSology-3.1-Release-Announcement-(Working-Version)

Key features in FOSSology 3.1 are:
Support for SPDX 2.1 document formats(tag:value format now available as well as RDF)

I have tried the fossology installation in the ways of docker as well as from source.
From the template (Only package/ document/file information) at https://github.com/fossology/fossology/tree/master/src/spdx2/agent/template, we could not generate SPDX 2.1 FULL features at https://spdx.org/spdx-specification-21-web-version

Is there any plans to use ALL features of SPDX 2.1 specification in future releases?
Thank you.

Regards,
Thuy Tran.

Release of 3.5.0

Shaheem Azmal M MD
 

Hello all,

After two release candidates, making fixes for REST API installation and various migration tests, FOSSology is stable enough for a new release. The main features of the 3.5.0 release can be found under RC1.

Particular corrections after RC1 can be found under RC2.

Mainly 3.5.0 adds more documentation, infrastructure improvements and support for brand new FOSSology REST API. A brief introduction about the REST API can be found at:

https://www.fossology.org/get-started/basic-rest-api-calls/

Moreover, new functionality has improved JSON output for nomos and restructured license detection for nomos. Last but not the least, FOSSology now have capabilities to ignore files specific to version control systems from the scanning improving scan times.

Credits

From the git commit history, we have following contributors since 3.4.0:

@ag4ums,
@ChristopheRequillart,
@AMDmi3,
@GMishx,
@mcieno,
@max-wittig,
@maxhbr,
@rlintu,
@sandipbhuyan,
@shaheemazmalmmd


Please find the release and binary packages for Debian and Ubuntu based systems here https://github.com/fossology/fossology/releases/tag/3.5.0 

Thanks & Regards
Shaheem Azmal M MD

Need to remove Debian packaging meta info from master branch

Gaurav Mishra
 

Hello all,

 

During our effort to publish FOSSology as a Debian package, we got few suggestions from the Debian community.

One of those suggestion is to remove the Debian packaging information (debian folder) from the master branch and put it into another branch like chore/debian/jessie.

 

This is done so to avoid conflicts as Debian maintainers will be editing this packaging information in the FOSSology mirror (hosted at Debian Sala). And any change in upstream can result in conflicts.

 

As this change will alter the packaging steps required by many of FOSSology users, we need your feedback.

 

I have opened an issue on GitHub for the same: https://github.com/fossology/fossology/issues/1341

 

Kindly respond either on this thread or on the GitHub issue if you have any concerns regarding the same.

With best regards,
Gaurav Mishra

GSoC 19 - Spasht Agent Documentation

vivek kumar
 

FOSSology 3.6.0 release

Shaheem Azmal M MD
 

Hello everyone,

After two release candidates, making fixes for migration tests, unified report and load issues with tree-view, FOSSology is stable enough for a new release. The main features of the 3.6.0 release can be found under

[RC1](https://github.com/fossology/fossology/releases/tag/3.6.0-rc1). Particular corrections after RC1 can be found under [RC2](https://github.com/fossology/fossology/releases/tag/3.6.0-rc2).

 

Few interesting features in this release are: 

 

  • A new agent named `ojo` (eye in Spanish) which does dedicated searches for the 'SPDX-License-Identifier' statements
  • Improved handling of manually added copyright statements to files
  • Improvements to the SPDX reporting, for example output also of comments
  • Calculating the SHA256 values for files from now on, because that is going to be used for integration of, for example, Software Heritage or Clearly defined

 

Credits to 3.6.0

 

  From the git commit history, we have following contributors since 3.5.0:

 

  > @andi8086 <andreas.reichel@...>,

  > @ag4ums <anupam.ghosh@...>,

  > @hastagAB <classicayush@...>,

  > @chienphamvu <chienphamvu@...>,

  > @ChristopheRequillart <christophe.requillart@...>,

  > @GMishx <mishra.gaurav@...>,

  > @maxhbr <maximilian.huber@...>,

  > @mcjaeger <michael.c.jaeger@...>,

  > @NicolasToussaint <nicolas1.toussaint@...>,

  > @PeterDaveHello <hsu@...>,

  > @rlintu <raino.lintulampi@...>,

  > @sandipbhuyan <sandipbhuyan@...>,

  > @shaheemazmalmmd <shaheem.azmal@...>,

  > @tiegz <tieg@...>,

  > @vivekaindia <vvksindia@...>

Please find the release and binary packages for Debian and Ubuntu based systems [here](https://github.com/fossology/fossology/releases/tag/3.6.0).


Regards,
Shaheem Azmal M MD

 
 

Google Summer of Code final report, Integration of Software Heritage in FOSSology

Sandip Bhuyan
 

Hello Everyone,
I am Sandip Kumar Bhuyan was a Google Summer of Code 2019 intern for fossology organization. Thank you for selecting my proposal for this year GSoC. I think I have reached the expectation. I was working on integrating software heritage in fossology. I have completed the GSoC 2019 successfully. You can find my report in my blogpost Blog Post. I hope everyone will like it.
Cheers
Sandip

--
Sandip Kumar Bhuyan | sandipbhuyan@... | sandipbhuyan.com
Have a great day| Code better