logo~stef/blog/

/dev/tags

fnords, net, hack, report, english, fun, personal, badtech, projects (h.a.c.k., longstrings, grindr, timecloud, tvhelyett, utterson, parltrack)

/dev/read

Watch videos at Vodpod and other videos from this collection.

Blogroll


generating pgp ids

2013-04-04

A proper fingerprint from Wikipedia The tool I release today is genkeyid part of my gpk PGP key management suite, which is a tool that helps you bruteforce arbitrary PGP key ids by modifying the timestamp field of public keys so that the packet hashes to a given key id.

I also release setfp.py which allows you to set arbitrary timestamps in PGP RSA key pairs and recalculates the RSA signature accordingly. You might want to combine this with the other already previously released genkey tools.

The two steps are separated, because the bruteforcing does only need a public key, but setfp also needs an unencrypted private key. So if you want to have a special key id, but also maintain Key Management Opsec, you should do the patching offline in a clean system that you discard later.

For the truly ignorant and the ones having extra clean systems and lots of entropy available in bulk, there's genid.sh which does the two steps in one, generating as many unencrypted keypairs as necessary until a suitable is found.

Of course this is nothing new, there are existing examples of manipulated key ids. Some people have issues with the ambiguity of key ids, but one of the authors of PGP says this is ok. The PGP FAQ has more on this.

get it from github/stef/gpk

Or read more: README.genkeyid.org


Announcing pwd.sh

2013-04-03

postits as password managers I wanted to switch to KeepassX to store all my passwords, but I wanted to use GPG to encrypt the passwords. So I came up with pwd.sh. It's a simple shell script that you can bind to your window manager keybindings, and when you invoke it, it uses the current focused window to deduce a key to store the user and the password. For better browsers like Firefox, Chromium, luakit and uzbl this means the currently loaded URLs, for all other windows the current window title. When creating a new password, it is automatically generated only the username is queried. I also wrote a small script that imports all passwords from Firefox into the new format. I'm very happy that now all my passwords isolated from my browsers and they are also protected by my PGP key on my external cryptostick.

When I showed this yesterday in our hackerspace, 2 members immediately installed and started massively improving pwd.sh, thanks asciimoo + potato!

So if you're running linux, like stuff based on the KISS principle, and are a crypto/gpg fetishist you might want to consider trying out this new "keepassx niche-killer" ;)

Check it out: pwd.sh


tlsauth

2013-04-02

certificate in firefox I just released tlsauth, a lightweight implementation of a CA and supporting scripts and config snippets that should make TLS client certificate-based authentication a bit easier to set up. The current implementation works in nginx (if someone knows how to do this in Apache, please contribute).

I also provide Flask-tlsauth and Django-tlsauth bindings, available also on pypi. Both contain simple web-based Certificate Authority functions, like sending in CSRs, listing and signing them, and even something similar to regular user registration. With the only difference, that when you are finished registering you have to import the certificate.

So when you look at this from a traditional PKI perspective something is fishy. User registration, and I get a cert back? Wait a minute, shouldn't the CSR be submitted by the user in the first place? Yes. But. :) Considering this from a traditional user registration workflow, the user usually trusts the server with his secret, the password. With TLSAuth however the server drops the secret after creating it and sending it to the user. So with most users blindly trusting their service providers I assume they'll trust them also diligently dropping them. The certs are not very good for anything else than log in to the server. And the CA can produce certs as many as he wants anyway.

Why is this good?

No more passwords

Your users win, because now they only need a password for importing the key into their browser, and then it is protected by the browser master key. This also prohibits users to reuse the same passwords on unrelated sites.

You can also copy your key around and load it on different devices, if you want to be able to access the services also from them, but this only needs to be done once in each browser.

This means also automatic authentication on all services sharing the issuing CA with the clients issuer. This means you can log in to all services on various servers certified by your issuing CA.

With appropriate security tokens you can even store your keys on smartcards and keep you certificates safe from your browser.

No more user databases!

Server operators win because they do not need to store a user database! This removes all kind of privacy issues, and reduces the costs of database leaks considerably.

Your users always send their their TLS cert, which is signed by the CA - you. So when someone comes and says: "hey i'm Joe, here's a certificate about that from you", then you can be sure about it. ;) Also a cert can contain more information, like an email address, or even an real life address for shipping, etc. You decide when you sign your users certificates what your require them to contain.

Authentication on TLS level

You know your client before it even says "GET / HTTP/1.1". This means you can redirect your handler accordingly, showing static only content for unauthenticated visitors, full dynamic server-side scripting and security bugs for trusted peers, and maybe even IMAP or SSH for certain certificates. ;)

Why is this bad?

Bad Browser UIs

On the user side log out is kinda impossible currently. But there seems to be a key-manager for stock firefox - iceweasel is not supported :/ - that could be helpful with log out and other key management related tasks.

It would be nice if the vendors would put more effort behind improving their related user interfaces instead of slacking or reinventing existing protocols.

Losing your phone/tablet/laptop

Losing HW is always a bad thing, especially when you have your certificates on it, hopefully they are protected by a master password in the browser, and full disk encryption on the hard drive. But this should be standard anyway.

Deleting users

CRL or OCSP (and OCSP Stapling already supported in nginx) are the normal way to do this. The question is how to keep track of the serial numbers without exposing the privacy of the end users by keeping server-side database.

Protecting your own CA root key

This is something that kinda makes the operator the weakest link in the whole setup. If anyone has access to your CA signing key, they can MITM attack any connections of all browsers that trust this CA. So you should apply utmost key management security with air gaping and possibly use some kind of cheap HSM like a smartcard or even better.

Loose ends

I understand that TLSAuth does not solve all problems. But for small groups or projects TLSAuth might make a lot of sense. It's perfect for protecting a phpmyadmin from all the probes on the internet, and still make it available to the admins, or you can run your own webmail for all your family and not care about the web as an attack vector.

There's a few open questions and loose ends to be explored here. But I'm quite hopeful to use TLSAuth in future projects, maybe even Parltrack.


Possible Parltrack features

2013-02-13

I've been maintaining a list of possible features for Parltrack if the funding campaign hits 10.000 EUR, I'd be interested to hear feedback and other suggestions to this list:

Monitor by subjects

Parltrack already provides listings by subjects (e.g. Protection of privacy and data protection) but there's neither a possibility to subscribe to any changes or new dossiers to these listings. Also missing is currently a user interface where users can browse and select all existing subjects. This feature would allow for broad tracking of policy areas instead of the currently supported dossier-by-dossier tracking.

Monitor by search phrase

Simply enter a search phrase and your email and get notified, if any dossier appears or changes that contains this phrase in its title.

Subscription management

A user-interface to better manage your subscriptions to things you're monitoring.

Visitor Trends

Display any trending dossiers or MEPs based on the visitor access statistics. This way you can identify what or who is currently hot in the EP.

Amendments from the 6th term

Adding also the amendments from the 6th parliamentary term between 2004 and 2009, different formats require the tuning of the scrapers to handle also these earlier documents.

Historical view

The preservation of historical data allows to present also snapshots from previous points in time. A nice timeline visualization is also imaginable.

Localized Parltrack data

Parltrack currently only scrapes in English, some information is easily scrapable also in the rest of the 22 European languages. Some might be harder, but for NGOs it would definitely make a difference, having this information also in their native language - especially if we're talking about re-users of the liberated datasets.

Commenting on dossiers and MEPs

Last but not least a feature that I have been long contemplating. It would be nice to somehow merge Pippi Longstrings, Herr Nilsson and Parltrack into a useful bundle, creating a possibility to comment on the legislative proposals and their procedural meta-information in one location. The issue with this is, that a public service like this needs a lot of moderation, and I fear that serious NGOs would not want to trust their internal political insights and commentary with an untrusted 3rd party like Parltrack. This feature is also the basis for the 750 EUR perk in the campaign by the way ;)

Conclusion

So this would be an initial list of medium to big features to be added, in addition to the site redesign and various small improvements that come up in the mean time, with possible other yet unplanned features to be added to this list. I expect this to occupy me for about a year especially if we reach funding levels that allow me to add new data sources as well.

There is also continued cooperation with NGOs reusing the Parltrack database, like with La Quadrature Du Nets awesome Political Memory and the just recently started Lobbyplag initiative which wants to expand its operations beyond the Data Protection dossiers.

If you agree with all or some of these goals, please consider supporting the current fundraising campaign by donating and making other people aware of this initiative. If you feel some important thing is missing let's talk about it, information and financial feedback are both important for the future of Parltrack, thank you.


parltrack2013

2013-01-29

EP - ACTA vote About two years ago Parltrack started as another tool trying to get some information that was necessary at that time. Since then the amount and quality of data in Parltrack has come a long way. One year ago, I had to rewrite all the scrapers as the European Parliament upgraded their website. A couple of related tools have been developed, for example Herr Nilsson or - the most widely-known - Political Memory or memopol as we call it. Also ACTA has been defeated. I believe Parltrack contributed a small part to this success. Having recent and good data on the ground was essential for campaigning in and around the European Parliament.

I think Parltrack is a tool with lots of potential. I'd really like to find some more time to just data-mine Parltrack, which was one of my initial motivations when I started this project. As a good friend used to say: most of our work in the commons is financed by pre-accumulated wealth from the traditional system. The peculiar nature of this open data combined with free software makes it somewhat difficult to keep this project sustainable. I've tried Flattr, debated and rejected advertising, offered consulting/custom development jobs, and turns out i'm too small to be eligible for EU funding grants. Depleting resources resulted in a shift of my attention lately to other jobs, however Parltrack seems to be used quite a lot. The lack of maintenance already started showing, so to stop this degradation and to allow me to focus more on Parltrack in the coming year I started an Indiegogo campaign. If you care about freedom, datalove, kittens, puppies, or just me, go here and support this campaign. It will allow me to build more free infrastructure.

thanks, s

Thank you to all my friends who helped me setting up this campaign.

ps: for Parltrack related news you can follow @Parltrack, and RSS updates


cyberfud

2012-11-28

src:http://guckes.soup.io/post/19675336/Fear-FEAR The usage of "cyber" as a prefix is a strong hint for lack of detailed knowledge into a certain topic, the intent to make a profit or take control by diluting the exact issues. Hiding the issues behind such muddled phrasing does not help the understanding and possible solutions.

The more often you hear "cyber" the stronger should be the sense of your "bullshit-meter" signal. Chances are high, that it's about spreading FUD to sell an oppressive and expensive security theater - cyberfud is for the internet, like the liquid-bomb was for airport "security".

So if this greed is only going to make us more oppressed, not safer then how to deal with all these menacing online threats that we hear about in the evening news?

A very wise man said[MP4 video]:

"...I'm suggesting, the internet itself can in no more meaningful sense be secure, than the oceans are secure. The security activities in the oceans, there's the "law of the seas", there are many aspects of it, but the functioning of humanity has depended on the openness and diversity of the seas and i think it depends similarly on the openness and diversity of the internet..."

There's a saying in software development: "a bug is cheapest, when caught as early as possible in the development process". Meaning it's cheaper to fix bugs during unit testing, than after they've been shipped to customers. So instead of starting an arms race to create expensive defensive snakeoil technology, we should focus on making the software more resistant. There's excellent examples, some critical infrastructure - our browsers - show a good understanding of this principle:

Compare this with Siemens not fixing the bug for 625 days that enabled the Stuxnet malware to operate.

It is irresponsible that a vendor waits 625 days to fix bugs that can affect critical infrastructure. Choosing the right words is important, forget cyberfud, here's a positive message:

Responsible Vendor

Closed-source vendors that have a consistent track record fixing bugs promptly and exercising diligence should be awarded, those who are not, should be penalized with full liability.

Instead of spreading cyberfud there should be a publicly available resource where users can check the security track record of vendors, vendors must be absolutely transparent about the vulnerabilities in their products, and it must be possible to objectively compare, measure and rate the vendors according to this data. Procurement decisions must be based on this as a obligatory condition: "no transparency and no sign of responsibility, no contract"

This idea of vendor liability is not new, hackers raised this issue already 14 years ago in a testimony before the US Senate.

I know, this issue cannot be solved solely by suddenly turning this industry into responsible vendors, among others problems are:

  • irresponsible customers disabling security features
  • restrictive laws outlawing security tools reduce the defensive capabilities of the network (like outlawing the immune system),
  • education, instead of paternalizing users into a victim role,
  • increased privacy awareness on the demand side and a strict adoption of the "data-minimization" principle would reduce the amount of "bounty out there" to grab.

The next time you hear about a cyberfud event, or hear some industry guy talking cyberfud, ask a few unsettling questions about commercial vendors externalizing the costs of security that are then exploited by greedy security-corporations and politicians. You are also free to ridicule:

"ich find ja, daß william gibson der einzige ist, der 'cyber' sagen darf, ohne dabei blöd auszusehen" — fx #alternativlos #ftw

(Translation: "The only person who is allowed to use 'cyber' without looking stupid is William Gibson".)


PGP key generation

2012-10-02

With the usage of PGP in everyday life our communication is mostly state of the art, and quite expensive to compromise. The weakest links are nowadays the systems where the communication terminates and is decrypted to plaintext. Not only are the messages available unencrypted, but also the encryption keys. Proper key management becomes essential, however diligent key management is something that not even the German Wehrmacht was always able to do properly. :) To reduce the probability of errors, there's a script at the end that automatizes most steps.

One essential aspect of key life-cycle management is key generation.

Note: most of the procedure below can be substituted by using an OpenPGP smartcards which allow to generate keys, that cannot be extracted easily, all signing and decryption happens in the smartcard itself. Such smartcards however usually have certain storage limits. Current technology allows usually 3 keys with 3072 bits, some newer models also 4096.

Generating a new key

Needed things:

  • a secure offline environment for key generation,
  • secure offline location to store the signing key,
  • Another secure offline location to store a backup of the signing key,
  • A third secure offline location to store a revocation certificate,
  • A pristine offline system for generation and handling of the key,
  • 3 distinct and strong passphrases

Offline system for key management

The biggest threat to key generation is a trojan/malware compromised system that leaks not only the keys but also captures the keystrokes of the passwords. To counter this threat it is strongly advised to use a pristine live CD to boot into an offline environment (yes, disconnecting the network cable is a good idea anyway). I like Tails for such a live system, but Privatix or Liberté Linux might be similarly useful.

Most PGP keys consist usually per default of a signing and an encryption key. The web of trust is woven by signing other peoples signing key. However there's a trade-off, either the key has a limited lifetime, and we have to ask our peers to re-sign the new key from time to time. Alternatively you create an unlimited signing key, but then handling becomes difficult as you want to protect this key with increased diligence. Multiple sources online suggest creating a master signing key, which is only used (again offline - see a pattern here?) for signing other keys and at least two subkeys, one for signing anything else and one for encryption. This allows you to regularly update your sub-keys without the need to re-sign them with your peers, only your master key needs to sign the new sub-keys. Regenerating your sub-keys regularly also reduces the lack of perfect forward secrecy in PGP.

All three keys should be protected by strong passphrases. As signing and encryption are not necessarily happening with the same frequency and applications, so it makes sense to set different passphrases for both. That means 3 passphrases in total, sounds hard to remember. Instead of using a long random password, rather use a passphrase consisting of at least 5 words. Now instead of 14 random letters you have to remember only five words, that should be manageable - especially if you make up a small story out of the words. One way to generate such passphrases is the Diceware method, another way is this simple script which replaces dices with openssls rand and the word list can be any size that fits in memory - this can be seeded with any kind of word list that you trust.

When you generate a key, it is good practice to also generate a key revocation certificate for the case your key gets compromised. In cases where you might not have the key available anymore to generate such a certificate having one ready up-front can prove useful. For cases when you lose or destroy the encrypted container containing the private keys it is also useful to have a backup ready - after all it took a lot of entropy to generate the key, don't waste it ;).

Both the revocation certificate and the backup are hopefully very rarely needed, but should be well protected. You can choose to use passphrases and encrypted containers, or you can encrypt the cert/backup with a 128 byte cryptographically strong random key and use Shamirs Secret Sharing Scheme to split up the encryption key into multiple parts and distribute them geographically and perhaps to trusted persons. Only when a preset amount of shares is presented, can the backup/cert be accessed. You could also generate a third set of shares for your backup, in case something happens to you and you want your family, friends or lawyers be able to read encrypted data belonging to a certain private key...

As you should very rarely need to generate such a full key and it is quite a complex procedure, there's a script that tries to automatize all the steps above. It depends on gpg, gpgsplit, srm, openssl and ssss, of which i think ssss might be necessary to install manually on tails. The script generates all interim material into /run/shm so that no trace on storage media is left, you have to move the various pieces yourself to their final location, like importing the subkeys into your keyring, distributing the shares for the backup and the revocation cert and the master signing key and it's backup copy. I will try to cover the storage of keys on dedicated USB sticks in a later post. I hope you enjoy your new pimped keys (oh and by the way nothing prevents you from having more than only two subkeys).

Comments and improvements are welcome.


amendments in parltrack

2012-09-24

Here's a sneak preview of an upcoming parltrack feature:

http://parltrack.euwiki.org/amendments/2011/0430(COD)

The data is possibly not complete, but gives good additional information. If everything goes well this will be integrated into MEP and dossier views. Until then you can change the dossier id in the url above, or replace it with a name of a mep:

http://parltrack.euwiki.org/amendments/SCHAAKE%20Marietje

Some stats on the data,

  • total number of amendments in the 7th term so far: 168917,
  • amended dossiers: 976,
  • amending MEPs: 775.

top 3 MEPS:

  1. Olle SCHMIDT: 2038
  2. Philippe LAMBERTS: 1974
  3. Silvia-Adriana ŢICĂU: 1610

top3 amended dossiers:

  1. 3075: Structural instruments: common provisions for ERDF, ESF, Cohesion Fund, EAFRD and EMFF; general provisions applicable to ERDF, ESF and Cohesion Fund (2011/0276(COD))
  2. 2482: Common Fisheries Policy (2011/0195(COD))
  3. 2310: Public procurement (2011/0438(COD))

If anyone wants to play with the raw data:

http://parltrack.euwiki.org/dumps/amendments.json.xz

And to see what data might be missing:

http://parltrack.euwiki.org/dumps/amendments.log


Tunnel daemons

2012-08-22

molerat Various methods of tunneling ssh connections to pierce through restrictive firewalls. The following setups are evaluated:

  • HTTPTunnel+stunnel4, moderately difficult to setup, but once installed it appears as legitimate HTTPS traffic.
  • Iodine, setup needs the most effort, however when done and the network allows DNS queries it works quite reliably.
  • CurveCP, setup is quite easy, when done the link is encrypted and fast. However the firewalls that allow UDP/53 to pass are somewhat limited.
  • ICMPTX, setup is quite easy, however there is no encryption, use it only to tunnel encrypted traffic like ssh and such.
  • TOR, setup is easy, usage is a bit delayed due to the latency of the Tor network, requests look like normal HTTPS traffic.

The setup with the most effort seems to be also the most reliable, an iodine based link over DNS can break out of a lot of networks. If we can use HTTP to browse but other services are restricted then the httptunnel is adequate. For less setup-hassle but increased latency Tor tunnels also deliver reliably. The usefulness of ICMP and CurveCP tunnels depend on the firewall configuration, but if they work, they're pretty fast.

Over HTTPS

This method is generally useful in heavily restricted networks, where you can only use the web for browsing, but no other services are allowed.

We use the fine tool httptunnel for masking our ssh connection. However httptunnel is not encrypted, and thus also the ssh handshake can be identified in the traffic. To avoid that, we put a tunnel into our tunnel using stunnel.

On the server

First we need to generate the SSL certificate:

openssl req -new -x509 -days 365 -nodes \
       -out htcert.pem -keyout htcert.pem

Set up an stunnel, make sure to set the ip address, and the user and group exist:

/usr/bin/stunnel -f -r 127.0.0.1:8888 \
     -d <public ip address>:443 \
     -p htcert.pem -s stunnel4 \
     -g stunnel4 -P ''

When this is done, we can run httptunnel to connect the sshd with the stunnel:

/usr/bin/hts -w -F 127.0.0.1:22 127.0.0.1:8888

On the client

Get the generated certificate from the server (don't forget to remove the private key part). You need to rename the cert to it's hash value and append a '.0':

mv htcert.pem $(openssl x509 -noout -hash -in htcert.pem).0

Now start the stunnel:

sudo stunnel -c -d 127.0.0.1:8888 \
       -r <server-address>:443 \
       -s stunnel4 -g stunnel4 -P '' -a . -v 3

We need to set the server address (can be IP or name-based), make sure the user, group exist.

start the httptunnel:

htc -F <httptunnel-port> 127.0.0.1:8888

The tunnel will listen on httptunnel-port. Enjoy your ssh-over-https:

ssh -p <httptunnel-port> 127.0.0.1

Over DNS

In some cases internet access is blocked however DNS traffic is allowed to pass, allowing us to tunnel through DNS.

If you can setup a special DNS entry for this, tunneling through DNS is very easy using the excellent iodine tool. Follow the straight-forward installation instructions.

Use this method if the network allows resolving of names, even if a local DNS server is forced on us, the tunnel will still work due to recursive queries hitting your "authoritative server".

Hint: you can manage and delegate a DNS zone for free on affraid.org, if you don't have your own.

Using CurveCP on UDP/53

The drawbacks of using any DNS protocol based tunnel like iodine, are that the tunnel has size-wise a huge protocol overhead, you need to setup a slightly uncommon DNS configuration and domain names you control are usually registered on your real name. If the firewall does not force the usage of a local DNS server and it allows traffic to UDP/53, then CurveCP tunnel is a preferred option.

Alternatively you could also run on UDP/80 or other allowed UDP ports.

On the server

Note: During testing I had to recompile CurveCP as the address family was missing from the bind call, see the patch at the end of this post.

Create a server key:

curvecpmakekey serverkey

convert the key to hex, and store it on the client in serverkey.hex:

curvecpprintkey serverkey > serverkey.hex

run the curvecpserver:

curvecpserver <your host name> \
                serverkey \
                <your ip address> \
                53 \
                00000000000000000000000000000000 \
                curvecpmessage /usr/sbin/sshd -i

On the client

This depends on socat the excellent swiss-army knife of socket handling.

Store the serverkey.hex that you generated on the server and run the client:

curvecpclient <curvecpserver hostname> \
    $(cat serverkey.hex) \
    <curvecpserver ip address> \
    53 \
    00000000000000000000000000000000 \
    curvecpmessage \
    -C sh -c "/usr/bin/socat tcp4-listen:9999,bind=127.0.0.1,reuseaddr,fork - <&6 >&7"

Start your ssh-over-curvecp:

ssh -p 9999 127.0.0.1

Over ICMP

Using ICMPTX you can set up tun devices that tunnel over ICMP, which is quite handy as in some cases it's not filtered and allows to pierce through the blockades. ICMPTX creates a local network device, so tunneling anything is quite easy after setup.

On the server

Simply run

(sleep 1; ifconfig tun0 10.0.99.1 netmask 255.255.255.0 )&; icmptx -s <server ip address>

On the client

Simply run

(sleep 1; ifconfig tun0 10.0.99.2 netmask 255.255.255.0 )&; icmptx -c <server ip address>

sshing to your box is then a simple:

ssh 10.0.99.1

Using Tor

Tor is great for hiding traffic, it's latency is a bit bigger than usual, but it's quite possible to get work done through tor tunnels even with ssh. If you configure your client-side tor proxy to use a tor bridge that runs on port 443, then the tunnel looks like casual HTTPS traffic.

There's two options, you can connect from a tor exit node to your normal ssh server, in this case skip the "On the server" part, and use your normal hostname instead of the .onion address referenced there.

On the server

If you want to run your ssh tunnel as a tor hidden service you simply have to add the following two lines

HiddenServiceDir /var/lib/tor/sshtun/
HiddenServicePort 22 127.0.0.1:22

to your /etc/tor/torrc. And find out the hostname of your new hidden service with:

cat /var/lib/tor/sshtun/

On the client

You simply need to call the torified ssh:

torify ssh <.onion hostname from server>

Code

Stubs for running the server-side daemons using the excellent runit tool can be found on github. These can be most easily installed using deamonize.sh. For client-side setup use the instructions in this post.

curvecp patch

curvecpserver had to be patched, as the address family in the bind call was uninitialized, the patch is below:

diff -urw nacl-20110221/curvecp/socket_bind.c nacl-20110221-new/curvecp/socket_bind.c
--- nacl-20110221/curvecp/socket_bind.c 2011-02-21 02:49:34.000000000 +0100
+++ nacl-20110221-new/curvecp/socket_bind.c     2012-08-19 02:52:25.000000000 +0200
@@ -9,6 +9,7 @@
 {
   struct sockaddr_in sa;
   byte_zero(&sa,sizeof sa);
+  sa.sin_family = AF_INET;
   byte_copy(&sa.sin_addr,4,ip);
   byte_copy(&sa.sin_port,2,port);
   return bind(fd,(struct sockaddr *) &sa,sizeof sa);

pippi matures

2012-07-15

New filtering interface for pippi I wanted to pippi CETA with ACTA, some other FTAs (Korea, Cariforum) and some other docs, but found it difficult to do so. So during the last days I revamped pippi a bit.

The results are a new browsing interface, where you can directly start pippifications of documents. Clicking there on the "Pippi ★" button uses the currently selected document and compares it with all shown starred documents. You can quickly filter all documents based on their title (this search uses powerful regular expressions), you can filter on your own documents (more on that later) or starred documents. Later is useful for running a pippi against a greater selection of reference documents.

Another new feature is that it is encouraged to be logged in when creating documents, this allows you to later edit the title of the document and to delete it if it is not yet pippied against other documents. Being a creator of a documents allows you also to access these documents more easily by using a filter for your own collection when browsing documents.

So pippification of CETA against all those other documents was easy:

  1. I created all the documents (e.g. i copy/pasted ACTA from Oct 2011 from http://www.euwiki.org/ACTA/Tokyo_oct2, and used a bunch of CELEX ids for documents available on eur-lex.)
  2. I went to "browse" and filtered on my own collection,
  3. I starred all the relevant documents for pippification,
  4. I selected from this list CETA, so that it is displayed in the Details
  5. I hit "Pippi ★" and after some delay I got the pippied results presented.

The result looks like this: http://pippi.euwiki.org/doc/ceta_ipr_2012feb

Hint: if you enter ACTA in "Filter by tag" in the top bar, then it hides the copies from the other documents...


digitális tudomány konzultáció

2012-02-19

Tavaly ősszel volt egy bizottsági konzultáció: "Consultation on scientific information in the digital age", erre beadtunk az FCForum és az EDRi kooperációjában egy véleményt (pdf). Januárban kijött az eredmény, és elég egybehangzó a vélemény. Ide kapcsolódik, hogy az Elseviert - a tudományos lapkiadók piacvezetőjét - elkezdték az akadémikusok bojkottálni, elegük van a kizsákmányolásból nekik is.

Végezetül pár kivonat az eredeti véleményből:

Our world has progressed from the economics of scarcity to an economy of abundance - at least when it comes to knowledge, information and data. This radical and ongoing shift is affecting all spheres of life, from the entertainment industry to public sector information. Scientific research is sadly an area where the fruits of this change have not begun to be harvested, despite the fact that the internet, which is the most important agent of change in this respect, was born in the research community. It is a precondition for meeting the EU's agenda to be a global leader in innovation to harness and nurture the generative nature of internet-enabled collaboration.

The role of Europe

We feel that policy formulation at the European level on access and preservation issues is essential to make progress on these issues and therefore agree strongly. This is for two reasons: first, scientific research was borderless even before the advent of the information age. Second, the legal frameworks surrounding issues of access and preservation have been to a large extent subject to legislative efforts at the European level. Although no effective harmonisation has come from the Copyright Duration Directive (93/98/EEC), Copyright Directive (2001/29/EC), the Database Directive (96/9/EC) and IPRED (2004/48/EC) , they do affect access and preservation issues to a large extent.

Moreover, we feel that the the following problems need to be addressed, primarily in order to be able to pursue Europe's ambitions in science, technology and sustainable economic development:

A majority of raw research data is not accessible to the scientific community as a result of database rights and/or other limitations. Or at least not accessible without strings attached. Examples of this are the results of clinical trials of new drugs. There have been several cases where early access to this data would have prevented harmful substances being prescribed (e.g. the Paxil and the Vioxx scandals)1.

The current model of the scientific publishing industry is fundamentally broken. Authors submitting articles to scientific journals are unpaid or even have to pay for publication ("author-pays" model). The editorial boards and the peer reviewers of scientific journals are effectively unpaid. The cost of printing journals and of their dissemination has dropped in the past decades. The price of scientific journals nonetheless keeps on escalating2. The profit margins of the scientific publishers are exceeding 35% now, while the general periodical publishing industry operates at a margin of less than 5%. According to financial analysts, no value is added by the scientific publishers that remotely justifies these excessive margins3.

We suggest than a comprehensive reform should include at least the following actions:

* Database protection should be abolished. Irrespective of any rights that preexisted the aforementioned Directive, there should be an harmonised EU rule that factual data is not eligible for copyright protection.

* Both the duration and the extent of copyright protection should be revised downwards. Any policymaking should take into account that reuse of information is essential for scientific progress.

* The EU should harmonise the transfer of copyrights from the original author to others. Such a transfer would have to be temporary and subject to compulsory registration.

* By extension, further expansion of IPR-enforcement powers through unfortunate directives such as IPRED should be curbed. In this vein ACTA should not be ratified by the EU since it can only worsen the situation in this regard. The chilling effects of excessive damages, provisional measures and injunctions in this field cannot be underestimated.

We also agree that co-ordinating existing initiatives in EU Member States would be an appropriate role for Europe.

Furthermore, we agree that Europe should be involved in supporting the development of a European network of repositories (online archives). In addition to this, we feel that online archives should use open standards as meant by the EIF 1.0 definition to the furthest extent possible, in order to foster genuine access to knowledge. Whenever possible, scientific information should be public, the same way legislation and jurisprudence are.

Finally on this point, we strongly agree that Europe should encourage universities, libraries, funding bodies, etc., to implement specific actions. A specific action should be that (European) funding of scientific research should be made contingent on a) unencumbered disclosure of both raw (provided that there are no privacy issues with raw data) and processed data and b) publication through open access scientific journals. Release early and release often, to borrow a mantra from the highly successful open source software development community, should be the credo of European research.

...we would like to stress that the current process of public funding of research by the EU is deeply flawed, but the problems are not unsolvable The biggest problem is that the areas of research that receive public funding are currently selected on the basis of framework programmes of half a decade ago. To quote a commentator in Forbes Magazine: ".. the system of awarding funds is insular, long winded and in no sense responsive to markets – these are five and six year plans, laying out innovation priorities from, say 2007 – 2013. Which areas of research should receive money is decided by the people who will bid for it and projects are assessed by people who are also applying. The impetus for change is dampened by the weight and self-serving nature of the system."4

An alternative avenue that is worth exploring is the use of competitions to solve specific scientific challenges. This model has been deployed successfully by DARPA and private actors like the X-prize challenges or the InnoCentive marketplace5.

Furthermore, user-driven innovation should be fostered. Examples of this phenomenon are so-called fablabs which put prototyping equipment in the hands of artists and designers. Even more grass-roots are hackerspaces, which despite their tremendous difficulties with housing and materials, have already shown to act as incubators for SMEs6.

1. Jasonoff, Transparency in Public Science, Purposes, Limits, in: Law and Contemporary Problems, vol. 69, iss. 21, Summer 2006, pp. 21-45.

2. see also Glenn S. McGuigan, Robert D. Russell, The Business of Academic Publishing: A Strategic Analysis of the Academic Journal Publishing Industry and its Impact on the Future of Scholarly Publishing, in: Electronic Journal of Academic and Special Librarianship, v.9 no.3 (Winter 2008)

3. http://southernlibrarianship.icaap.org/content/v09n03/mcguigan_g01.html

4. http://www.forbes.com/sites/haydnshaughnessy/2011/07/11/europes-disintegration-its-not-about-the-piigs-or-the-euro/

5. http://www.innocentive.com/

6. An example of this is innovation award winning soup.io


hacktivism hour cccamp2011

2011-08-28

At the camp we had a very inspiring radio show [ogg]. One interesting topic that came up was copyright abolitionism. As a free software developer it's hard for me to accept the loss of protection by the GPL from closing down free software. I can agree however, that we must change our discourse from copyright to alternative ways of incentivising value creation. If we can replace copyright with alternative systems that empowers creators and amplifies creation while somehow also preserving the 4 basic rights of free software, I'm all OK with that. (I admit the mixing of free software ideas with more general value-creation, needs some more refinement, but you get the idea.)


omnom-announcements

2011-08-26

I experiment with omnom to use it also as a platform to announce updates to itself, check out the omnom-announcement tag or point your rss reader at the atom feed.

The latest message in a nutshell: for anyone who uses the userscript for bookmarking should update.


widget for omnom

2011-08-24

Good news everyone! Omnom - my feeble attempt at creating a proper^Wlibre delicious replacement - now has gotten "widget" functionality. I took the original delicious widget and shamelessly adopted it. You can see the result in the right bar under "/dev/read".

If you are one of the lucky omnom users, you can use the code below, just change the 2 links pointing to my collection to your own.



<h3><a href='http://links.ctrlc.hu/u/stf'>/dev/read</a></h3><div id="omnom-box" style="margin:0;padding:0;border:none;"> </div>
<script type="text/javascript" src="http://links.ctrlc.hu/u/stf/?format=json&j"></script>
<script type="text/javascript">
   var ul = document.createElement('ul');
   for (var i=0, post; post = omnom_posts[i]; i++) {
      var li = document.createElement('li');
      var a = document.createElement('a');
      a.setAttribute('href', post.url);
      a.appendChild(document.createTextNode(post.title));
      li.appendChild(a);
      ul.appendChild(li);
   }
   ul.setAttribute('id', 'omnom-list');
   document.getElementById('omnom-box').appendChild(ul);
</script>



Announcing Parltrack

2011-07-11

cc licensed ( BY NC ) flickr photo by *TheLibertine*: http://flickr.com/photos/17cherrytreelane/3003973154/ Parltrack is a free tool and the associated free database to track the law-making in the European Parliament. It

  • aggregates and publishes dispersed information,
  • provides email and RSS based notifications,
  • links most documents with Pippi Longstrings for analysis and commenting.

Combined information

Parltrack collects the data daily and combines the information on MEPs, progress on dossiers from OEIL, committee agendas, and vote results from the plenary minutes. Using this data it is easy to see for example at a glance:

  • which MEP has taken responsibility for which dossiers,
  • which committee is responsible or gives opinion on which dossiers,
  • forecasts for next steps on dossiers in the EP and committees - including tabling deadlines for amendments,
  • online detailed committee agendas,
  • online calendar with EP and committee dates,
  • who are the influential MEPs related to a dossier. There's much more possibilities in this data than the web-interface and your humble developers can ever present to you. Though we are willing to work for hire to dig deeper into this information-goldmine. ;)

Notifications

how to subscribe Anyone can subscribe to receive email notifications on changes on dossiers. On every dossier page, there's a link "Track this Dossier", by providing your email here you create a new notification group. If you don't specify a name of the notification group, one will be randomly generated for you. If you specify an existing notification group name, the current dossier will be added to that. After you created a notification group, you can share the link to this so other people can join in.

The Texts

Most of the documents created in the law-making process (initial, supplementary and final texts, opinions) are linked whenever available from OEIL. Most of these documents are also available on Eur-Lex (the official EU website publishing the texts) and thus are also automatically available in Pippi Longstrings from the Documents tab for dossiers. Using Pippi Longstrings you can analyse the texts and comment on them. Commenting happens by selecting the text you want to comment on. The results of pippi-analysis are also displayed as comments, from a user named Pippi Longstrings.

The Data

All data in Parltrack is available freely in various forms for anyone to use. Most of the pages have a "Download as JSON" link, all new and changed dossiers are listed in an RSS feed, changes in dossiers are also available as email notifications. Furthermore a complete DB dump as JSON is also available. All this is under the ODBLv1.0 license. Running your own instance of Parltrack is encouraged, the code is free under the AGPLv3+.

Searching

Currently search is possible for text in MEPs names, dossiers titles and IDs of final acts like "directive 2006/24/EC".

Credit

Parltrack is inspired by ehjs Tratten, was conceived during the 2010 FCForum Tools for policy reformers workshop and is influenced by a bunch of good people, thanks!

If you'd like to support similar tools in the future, don't hesitate to show your support:

Thanks, It's much appreciated.



< prev posts

CC BY-SA RSS Export
Proudly powered by Utterson