Deploy smart contracts on Filecoin’s Virtual Machine →
EN 中文
Category:
Feb 14, 2019

Filecoin 2018 Q3 & Q4 Update: Development and Research Repos, Roadmap Update, Development Update, and More

Table of Contents

  1. Opening up development and research repos
  2. Project Roadmap Update
  3. Development Update
  4. Research Update
  5. Community Update
  6. Legal Update
  7. Filecoin Public Talks & Media
  8. libp2p update for Filecoin
  9. IPFS update for Filecoin
  10. Cautionary Notes

1. Opening up development and research repos

Good news! After many months building things privately, we are ready to open up a number of codebases and community forums. This is right on schedule as predicted in our Project Roadmap and last update. Read our announcement blog post for more.

2. Filecoin Project Roadmap Update

In our last big update, we published a high-level roadmap, with upcoming optimistic milestones.

We are pleased to report that – so far – we have hit our first two big milestones on time 🎉🎉, and added a third 🎉:

  • We started a go-filecoin collaborator & contributor preview in 2018 Q4.
  • We opened up the go-filecoin codebase in 2019 Q1.
  • We launched the first public Filecoin devnets in 2019 Q1.

Beyond that, our team has been hard at work in the following areas:

  • Developing go-filecoin. The main effort to implement the Filecoin Protocol. This is the bulk of our development work.
  • Developing rust-fil-proofs. Implementations and benchmarking of Proof-of-Replication and Proof-of-Spacetime algorithms, for use in go-filecoin, other filecoin implementations, and other systems.
  • Writing and refining protocol specs. We need detailed protocol specifications in order to have multiple independent implementations of the protocol, to have a properly decentralized network, and to assess and verify protocol security.
  • Ancillary product development. There are a couple dozen ancillary products and tools that need to exist around the Filecoin network: things like chain explorers, network stats visualizers, integrations, and so on.
  • Designing protocol improvements. There are multiple areas of the filecoin protocol that have improved significantly – in particular chain scalability, performance of proofs, and consensus security.

As we make progress, our visibility and estimates grow clearer. To that end, we have adjusted the upcoming milestones as follows:

| — | — | — | | Milestone | Old ETA | New ETA | | Launching the first public Filecoin testnet: | Q1 | Q2 | | Testing with sets of large scale miners: | Q1 | Q2 | | go-filecoin v1.0.0 feature freeze: | Q1/Q2 | Q2/Q3 | | Security Review & Audit: | Q1/Q2 | Q2/Q3 | | Launching the Filecoin mainnet: | Q2/Q3 | Q3/Q4 |

We have updated the Roadmap document accordingly. Please recall that all our estimates and timelines are optimistic and subject to change. We want to keep the community updated, and give a rough indication of where we are, what’s next, and when we currently (if optimistically) expect to achieve our next milestones.

3. Filecoin Development Update

This update, we are splitting the news into:

  • demos - new demos shipping today!
  • go-filecoin - the first implementation of the Filecoin protocol
  • rust-fil-proofs - a portable implementation of Filecoin proof systems and other cryptographic components.

3.1. Dev Update / demos

We are publishing 2 demos along with this update. The demos show a few of the ways you can get started with testing Filecoin proofs-of-replication (PoReps). PoReps make it possible to ensure that a Filecoin storage miner is storing their clients’ data correctly. They are crucial to the successful operation of the protocol. In the future, we will release more demos of features, go-filecoin installation, and how to use the newly opened devnets and related development tools, such as the network statistics dashboard.

Demo 1: Proof Benchmarks

In this demo, we show how you can run Proof-of-Replication (PoRep) benchmarks on your local machine. You can use these benchmark tools to observe how different parameters affect data replication and PoRep generation and verification times. You can also use these tools to benchmark any variations on implementations of PoRep. The demo shows:

  • cloning rust-fil-proofs and building the benchmarks
  • how to run the benchmarking tool for ZigZag-PoRep (our current PoRep implementation)
  • generating benchmark running times for ZigZag-PoRep on different data sizes: 1MB, 2MB, and 1GB

Demo 2: The Replication Game

In this demo, we describe the Replication Game and how to compete. We show how you can create and submit an entry to the Replication Game, and how to check your results on a leaderboard. More specifically, we show how to:

  • parse the different parameters that can be passed to the game
  • run a play script to generate a Replication Game submission from your local machine or rig and post it to the game server
  • check the Replication Game leaderboard
  • filter Replication Game leaderboard using the same parameters used to generate your submission

3.2. Dev Update / go-filecoin

  • ✅Completion overview updated. The roadmap document has a section tracking go-filecoin’s completion. We have updated this section with the latest state. Many components advanced from “still todo”, to “work in progress”, and to “mostly done”. We have also added and removed some components.
  • ⛽️Implemented gas for VM execution. Filecoin actors (the “VM” programs on Filecoin) require a concept of gas to run, similar to Ethereum’s. This key functionality landed in go-filecoin in the last few months.
  • 💎Miner Payments and Payment Channels. Miner payments are now implemented, with the use of payment channels. The latter ones are a key feature in making the Filecoin network operate faster (off chain protocols), and to scale to an acceptable performance (amount of chain storage per op). The mining cycle is close to complete.
  • 🕸Developer networks. We have reached the point where persistent networks are useful for development, debugging, and testing. We are not close to the proper “testnet” stage, but we already have the need for several small networks our developers, early contributors, and prospective users can work with. Today, we are introducing “devnets”, smaller development networks for experimental usage. Read more here.
  • 🗃End-to-end Data Storage. Storing data in go-filecoin is now working end-to-end. Miners (a) pledge storage with collateral, (b) make deals with clients, (c) receive clients’ data, (d) assemble the data into sectors, (e) seal the sectors with Proof-of-Replication, (f) gain storage power through sealed sectors, (g) mine on top of sealed sectors, (h) receive clients’ retrieval requests for data, (i) unseal sectors to retrieve clients’ data, (j) serve the data to clients, and (k) and get paid for doing so. From here, we will continue to improve storage performance and usability.
  • 🛠Turning prototype code into production code. Early on, in building systems like go-filecoin, we need to move fast and prototype. We have now moved on to maturing the codebase with principled software engineering practices, clean and clear interfaces, separation of concerns into components, layered APIs, and more. This has been a significant endeavor, key for long-term maintainability, new contributor onboarding, building on go-filecoin, security, and a smoother developer experience.
  • 📄Documentation and Guides. We created detailed documentation for getting started with go-filecoin, for building the projects from source, for connecting to devnets, for storage mining, for storing and retrieving data, for the network dashboards, for the block explorer, and for troubleshooting issues. We also wrote up an overview of go-filecoin’s code.
  • 🎉Got ready for new contributors! We structured our contributing guidelines, and documented our development processes. We made our issue tracker approachable to newcomers, with lots of issues with help-wanted or good-first-issue labels. We refactored and cleaned up APIs. We created #fil-dev and other developer chat channels (on Slack/Riot/Matrix). And we got our first pull-requests from outside the Filecoin team! 🎁

3.3 Dev Update / rust-fil-proofs

  • ⚙️Portable fil-proofs effort. Implementing Proofs-of-Replication (PoReps), Proofs-of-Spacetime (PoSTs), and other cryptographic constructions is a significant feat of engineering. In Q3, we decided to split this into a separate endeavor from go-filecoin, in order to (a) increase focus for each effort, (b) work closely with our cryptography researchers in “research-development” mode, and (c) produce an implementation-agnostic library. We decided to write it in Rust, which is a good fit to couple closely with other existing crypto libraries we need (eg Bellman), and to produce code and tooling that is usable to all implementations of the Filecoin Protocol and across a wide variety of systems and devices.
  • 🏃‍♀️DRGs, ZigZag-PoRep, and optimizing implementations. The main task of the rust-fil-proofs effort is to implement practical and usable PoReps and PoSTs. To achieve this, we needed to implement brand new cryptographic constructions (fresh from the research press), study and optimize the performance of each proof system, and produce tooling that go-filecoin now imports. We discuss the effort in more detail in these research interviews.
  • ⏱Reaching practical runtimes. PoRep algorithms are meant to be arbitrarily tunable in speed. However, to greatly reduce the size of proofs on chain, we needed to employ constructions that enabled non-interactive and aggregated proofs – namely, verifiable computation systems. In our case, the latest PoRep algorithms use zkSNARKs, and are quite slow. In Q3, we implemented algorithms and got the Filecoin Proving Subsystem working end-to-end. During Q4, we managed to speed up the runtime of PoRep replication by orders of magnitude! Sealing a 1GB sector took ~150hrs in Oct, and we optimized it to 50hrs in Nov, then 16hrs, then just under 1hr in Dec. 🎉 This is right in our acceptable zone, which is great.
  • 🔬Benchmark Tools. We developed benchmarking tools for a number of proof algorithms and cryptographic components. These can be used to measure the real, practical performance of algorithms with some given operations. These tools are great to have, and can be seen in action in this demo.
  • 👾The Replication Game. Last but not least: we made improving replication algorithms into a game! Players in the Replication Game can compete to outperform the default implementation of Proof-of-Replication, by optimizing algorithms, running on better hardware, and so on. Think you can optimize better than the rest? Prove it by beating others in the leaderboard. This is a fun & rewarding way to participate in building Filecoin. Some results may even beat those generated by our current PoRep implementation. We can’t wait to see what you come up with! See how to make a submission to the Replication Game in this demo.

4. Filecoin Research Update

A lot of important improvements have happened in our research team.

Research Interviews and Talks. In an effort to make our work more accessible, we are making a set of interviews with Filecoin researchers and engineers. Check out the first installment of these interviews:

Proof-of-Replication constructions. **The best Proof-of-Replication (PoRep) we have so far is ZigZag-PoRep, introduced in the middle of 2018. During Q3 and Q4, we implemented ZigZag-PoRep, and other PoRep algorithms. Bringing complex cryptographic constructions from theory into practice is a significant feat. Finding practical and application constraints that drive improvements in the theoretical construction is even greater. We achieved this and more with ZigZag-PoRep. We are also setting proof parameter choices based on practical limitations. The research work on PoReps led to novel academic results on Tight Proofs of Space and Replication.

Expected Consensus formalization and implementation. One of the key components of Filecoin’s *Storage Power Consensus *(SPC) is a protocol named Expected Consensus (EC). EC is a protocol inspired by leader-election based proof-of-stake protocols. In the last two quarters, we formalized EC’s properties and studied it in a variety of scenarios. We explored a number of questions through simulation, such as parameter choices, weighing function choices, and incentive compatibility. Significant contributions resulted from this exploration, that changed protocol implementation. Some of this work is of independent interest, and may be published separately. EC is now implemented within go-filecoin’s SPC implementation, and can be seen keeping the Filecoin blockchains stable across our devnets.

VDFs in Proofs-of-Spacetime, and VDF Collaboration. PoReps are not the only proof protocols in Filecoin that improved recently. We also have new constructions for Proofs-of-Spacetime (PoSTs) using recent Verifiable Delay Function (VDF) constructions. This is a significant improvement, as VDFs give PoSTs much stronger timing guarantees than we get from the bare PoReps. This is great because it allows us to have longer PoST verification cycles, and thus to significantly reduce the amount of data accumulated on-chain. This new direction has led to a collaboration with the Ethereum Foundation to drive development, implementation, and optimization of VDFs. Some of Ethereum’s new constructions aim to use VDFs within consensus, a direction Filecoin is also exploring. Ethereum, Filecoin, and a number of next-generation blockchain protocols will benefit greatly from this work. But perhaps the most exciting part is this could yield efficient, world-wide, decentralized, and robust randomness beacons – which would be a major breakthrough in applied cryptography and distributed systems, with wide applicability beyond blockchains.

RFP: Single Secret Leader Election. Over the last couple of years, an important problem has surfaced through our exploration of consensus protocols for Filecoin.. Its solution stands to improve random leader election based protocols, such as Filecoin and many other blockchains, particularly Proof-of-Stake systems. We have called this problem Single Secret Leader Election (SSLE), and it involves electing at most one leader, in a fair, unpredictable, and secret way, from a weighted set of participants. We have posted a new Request For Proposals (RFP) to solve this problem: RFP-6 Single Secret Leader Election. A solution to this problem would constitute a major breakthrough in cryptographic sortition schemes and greatly improve the security, performance, and simplicity of dozens of blockchain protocols, including Filecoin.

5. Filecoin Community Update

We have been preparing to meet the needs of a broad and vibrant Filecoin community. Three examples of this include launching the Filecoin Private Preview, supporting Global Filecoin Meetups and collaborating with leading research, engineering, and blockchain organizations.

  • Filecoin Private Preview. Over the last few months, we invited external research and developer collaborators to test drive the codebases released recently, and our community communication channels. Among other things, this helped us generate or improve the following community resources:
  • Filecoin Meetups and Communities. The Filecoin team is delighted to engage with and support communities and meetups around the world. These groups come together to learn about, discuss, participate in, and evangelize the Filecoin Network. If you’d like start a Filecoin meetup group, please email team@filecoin.org with your information and we’ll try to help however we can. A few specific events to highlight include:
    • Meetups. Filecoin meetups occur around the world on a weekly basis. For example, in 2018, around 90 meetup events have been held across 16 cities in China, Hong Kong, and Taiwan (we note this number reflects activity from communities that have specifically reached out to our team). These meetups regularly attract 100-200 attendees and are often formatted as lectures, panels, and developer salons. Online educational streams are also common and generate thousands of views.
    • Universities, Research Communities, and Research Centers. Community leaders have visited research centers, presented to blockchain student groups, and visited renowned universities.
    • Protocol Labs Events. Last year, Protocol Labs hosted its first LabDay where the stakeholders in PL’s ecosystem presented, hacked, and worked together. We delivered a presentation about active Filecoin Communities in China and moderated a panel that with a few of the community leaders there.
    • Other. Some community volunteers are actively working to educate local governments about distributed technology. Further, our community members have contributed translations, visited exchanges, hosted conferences and dev workshops, developed educational curriculums, and even baked filecoin mooncakes!
  • Filecoin Collaborators. We continue to develop deep relationships with leading research, engineering, distributed systems, and blockchain organizations, including:

The Permissive License Stack. Protocol Labs has developed and adopted a set of open source intellectual property commitments we call the Permissive License Stack. These commitments guarantee that the technology PL is building for Filecoin will be usable by the largest possible number of people and projects.

We’re hopeful that sharing our approach will help our colleagues at other companies and organizations to join us and follow this path forward. You can read more about the Permissive Licensing Stack here.

Engaging with policymakers about the decentralized web. Protocol Labs is a founding member of the Blockchain Association, a new policy advocacy group whose work we believe will advance the goals of the decentralized web. The Blockchain Association has done incredible work educating policymakers, providing critical industry knowledge to improve draft legislation, and promoting productive engagement between the U.S. government and important decentralized software projects. Their writings about decentralization, a recently proposed law in Congress, and a potential administrative law case may interest those following US public policy.

We also support Coin Center, a nonprofit think tank in Washington, DC educating policymakers on the promise of blockchain technology and why it should remain free and open. Our team provided some comments on their important new paper, The Case for Electronic Cash: Why Private Peer-to-Peer Payments are Essential to an Open Society.

In September, the General Counsel of Protocol Labs testified alongside 100 other industry experts to at a briefing hosted by two Congressmen who later introduced a strongly pro-cryptocurrency bill.

7. Filecoin Public Talks & Media

We elected to continue staying out of the public sphere in 2018 and focus on building. That said, our team and our broader Filecoin community gave a few public talks, published a few blog posts and visited a few research groups. Our activity will remain similarly constrained until the lead-up to the launch of the network.

Additionally, we ran a small conference on consensus protocols called ConsensusDay. This is the first installment of what we expect to become a regular event. Consensus throughput is one of the major hurdles in blockchain scalability, and we will only realize the full potential of crypto networks if we can scale operations to match the needs of typical consumer-oriented applications. We need to move orders of magnitude beyond the 7-25 tx/s (that’s Bitcoin and Ethereum) to billions of transactions per second. We believe this improvement will come from a combination of off-chain scalability protocols, sharding, hierarchy, and new directions in consensus protocols. We hope that ConsensusDay can stimulate the field, and draw attention to important problems. All the talks from the first ConsensusDay can be found here.

8. libp2p update for Filecoin

Filecoin is built on top of and depends heavily on libp2p, a project led by Protocol Labs as well. Therefore we include a compressed update from that project tuned for Filecoin.

9. IPFS update for Filecoin

Filecoin is built on top of and depends heavily on IPFS, a project also led by Protocol Labs. Many users of IPFS aim to also become users of Filecoin, so IPFS capabilities and use cases greatly inform Filecoin. Therefore, we include a compressed update from that project tuned for Filecoin.

10. Cautionary Notes

  • We are not announcing the testnet or the mainnet. Instead, we are inviting you to review and contribute to this open source work we believe is so important.
  • The three devnets are true development networks, and we make no guarantees about performance, stability, and security on the devnets. In fact, we can probably guarantee the opposite.
  • None of the economics are live and any tokens involved are only for testing. It is not yet possible to claim, sell, or buy actual tokens.
  • Miners cannot earn actual block rewards.
  • Exchanges claiming to trade in filecoin or filecoin futures are either fraudulent or very confused. There should not yet be any sales or purchases of devnet filecoin – or any other filecoin.
  • Nobody should invest in mining hardware based on any specs in the devnet, as these specs will likely change.
  • Beware of any potential scams anywhere based on claims contrary to the above. We have unfortunately heard of some potentially fraudulent activity in China and Hong Kong. Please review the Anti-Fraud Statement, which strongly recommends avoiding anyone in China or Hong Kong claiming to be commercially or officially affiliated with the Filecoin project (or IPFS or Protocol Labs) at this time.

Many Thanks!

Thank you for all your support so far, and in the future! And thank you in advance for your referrals, ideas, and suggestions. Your help accelerates our progress.

The Filecoin Team

Protocol Labs

Related Articles

View all articles
updates

Wolfram Blockchain Labs Expanding Storage Functionality with IPFS & Filecoin for Dapp Development

WBL is integrating their distributed ledger technology platform with IPFS and Filecoin to add new decentralized functionality and expanded storage to their existing blockchain features.

May 12, 2021

updates

Proof-of-Replication, Power-Fault-Tolerance and Research Roadmap

At Protocol Labs we are charting a course to the future, but we also know that the details make or break the plan.

Jul 27, 2017

Filecoin-Features updates

What sets it apart: Filecoin’s proof system

With the testnet launch, a new set of storage-based proof systems to attain decentralized consensus has been launched.

Feb 18, 2020

Filecoin is an open-source cloud storage marketplace, protocol, and incentive layer.
icon_client
filecoin_request_icon
filecoin_data_icon
filecoin_data_icon_black
icon_miner
icon_miner_other
filecoin_data_icon_black