Designing Nice Challenges for Cybersecurity Competitions

Not too long ago, the Division of Homeland Safety (DHS) recognized the necessity to encourage hands-on studying by means of cybersecurity competitions to handle a scarcity of expert cyber defenders. Likewise, in 2019, Govt Order 13870 addressed the necessity to establish, problem, and reward america authorities’s finest cybersecurity practitioners and groups throughout offensive and defensive cybersecurity disciplines. Effectively-developed cybersecurity competitions supply a approach for presidency organizations to meet that order.

The Software program Engineering Institute (SEI) has been working with the DHS Cybersecurity & Infrastructure Safety Company (CISA) to deliver distinctive cybersecurity challenges to the federal cyber workforce. This weblog publish highlights the SEI’s expertise growing cybersecurity challenges for the President’s Cup Cybersecurity Competitors and general-purpose pointers and finest practices for growing efficient challenges. It additionally discusses instruments the SEI has developed and made freely out there to assist the event of cybersecurity challenges. The SEI technical report Problem Improvement Tips for Cybersecurity Competitions explores these concepts in larger element.

The Function and Worth of Cybersecurity Challenges

Cybersecurity challenges are the center of cybersecurity competitions. They supply the hands-on duties rivals carry out as a part of the competitors. Cybersecurity challenges can take a number of kinds and may contain completely different responses, reminiscent of performing actions on one or many digital machines (VM), analyzing numerous kinds of recordsdata or information, or writing code. A single cybersecurity competitors would possibly comprise a number of completely different challenges.

The purpose of those cybersecurity challenges is to show or assess cybersecurity expertise by means of hands-on workouts. Consequently, when constructing challenges, builders choose mission-critical work roles and duties from the Nationwide Initiative for Cybersecurity Training Workforce Framework for Cybersecurity (NICE Framework), a doc revealed by the Nationwide Institute of Requirements and Know-how (NIST) and the Nationwide Initiative for Cybersecurity Careers and Research (NICCS). The NICE Framework defines 52 work roles and offers detailed details about the precise information, expertise, and skills (KSAs) required to carry out duties in every.

Utilizing the NICE Framework helps builders focus challenges on essential expertise that finest signify the cybersecurity workforce. Every problem clearly states which NICE work position and duties it targets. By figuring out the information and expertise every problem targets, rivals can simply give attention to challenges that handle their strengths in the course of the competitors and isolate studying alternatives when utilizing challenges for coaching.

Problem Planning

Creating profitable cybersecurity challenges begins with complete planning to find out the extent of issue for every problem, assessing the factors out there for every problem, and figuring out the instruments required to unravel the challenges. By way of issue, competitors organizers need contributors to really feel engaged and challenged. Challenges which might be too straightforward will make extra superior contributors lose curiosity, and challenges which might be too exhausting will frustrate rivals. Competitions typically ought to embrace challenges which might be appropriate for all ranges—newbie, intermediate, and superior.


Factors techniques are used to reward rivals for the effort and time they spend fixing every problem. Furthermore, competitors organizers can use factors to find out competitor placement—rivals with greater scores can advance to future rounds, and organizers can acknowledge these with the very best factors as winners. Factors needs to be commensurate with the issue posed by the problem and energy required to unravel it. Level allocation generally is a subjective course of, a matter we’ll return to within the part Problem Testing and Assessment part beneath.

Problem Tooling

Figuring out the instruments required to unravel a problem is a vital step within the growth course of for 2 causes:

  • It ensures that problem builders set up all required instruments within the problem surroundings.
  • It’s good apply to offer rivals a listing instruments out there within the problem surroundings, particularly for competitions during which organizers present rivals with the evaluation surroundings.

Builders needs to be cautious to construct challenges that don’t require using paid or licensed software program. Open supply or free instruments, purposes, and working techniques are important as a result of some rivals may not have entry to sure software program licenses, which might put them at a drawback and even forestall them from finishing altogether.

Problem Improvement

Builders have to be well-versed in cybersecurity material to plan revolutionary approaches to check rivals. Not solely should builders establish the abilities the problem will goal and the situation it would simulate, they need to additionally develop the technical features of the problem, implement an automatic and auditable grading system, incorporate variability, and write documentation for each the testers and the rivals.

Pre-Improvement Concerns

Builders ought to start by figuring out the work roles and expertise their problem goals to evaluate. By so doing, they’ll construct extra exact challenges and keep away from together with duties that don’t assess relevant expertise or that take a look at too huge an array of expertise. After they’ve outlined the work position related to a given problem, builders can kind a problem thought.

The problem thought contains the technical duties rivals should full and the situation during which the problem situation will happen. All problem duties ought to resemble the duties that professionals undertake as a part of their jobs. Builders are free to be as artistic as they want when constructing the situation. Topical challenges primarily based on real-world cybersecurity occasions supply one other approach so as to add distinctive and inventive situations to challenges.

Technical Part Concerns

The technical elements of problem growth typically contain VM, community, and repair configuration. This configuration ensures the problem surroundings deploys appropriately when rivals try the problem. Improvement of technical elements would possibly embrace:

  • Configuring VMs or companies to include recognized vulnerabilities
  • Configuring routers, firewalls, companies, and so on., to the state builders need
  • Staging assault artifacts or proof all through networks or logs
  • Finishing different actions that put together the surroundings for the problem

Builders may additionally purposefully misconfigure features of the surroundings if the problem targets figuring out and fixing misconfigurations.

Greatest Practices for Growing Challenges

Every problem targets completely different expertise, so there is no such thing as a normal course of for growing a cybersecurity problem. Nonetheless, builders ought to apply the next finest practices:

  • Make sure the technical expertise assessed by the problem are relevant in the true world.
  • Make sure the instruments required to unravel the problem are free to make use of and out there to the rivals.
  • Make a listing of the instruments out there to rivals within the hosted surroundings.
  • Guarantee challenges don’t drive rivals down a single resolution path. Opponents ought to have the ability to remedy challenges in any life like method.
  • Take away pointless hints or shortcuts from the problem, together with command historical past, shopping information, and different information that would permit rivals a shortcut to fixing the problem.

Problem Grading

Basically, builders ought to automate grading by means of an authoritative server that receives solutions from the rivals and determines what number of factors to award the submission. The submission system ought to typically ignore variations in capitalization, white house, particular characters, and different variations which might be in the end irrelevant to correctness. Doing so ensures rivals aren’t unfairly penalized for immaterial errors.

Ignoring these errors might sound to contradict an evaluation of operational readiness in instances the place precise precision is required. Nonetheless, cybersecurity competitions have objectives and issues past evaluating operational proficiency, reminiscent of guaranteeing a good competitors and inspiring broad participation.

Builders could apply completely different grading strategies, together with the next:

  • Token discovery. In token-discovery grading, rivals should discover a string or token that follows an outlined format (these tokens will also be known as “flags”). Builders can place the token in any a part of the problem the place the competitor will discover it by finishing the problem duties.
  • Query-and-answer issues. For question-and-answer issues, the competitor should discover the right reply to a number of questions by performing problem duties. The solutions to the problem questions can take a number of kinds, reminiscent of coming into file paths, IP addresses, hostnames, usernames, or different fields and codecs which might be clearly outlined.
  • Surroundings verification. In surroundings verification grading, the system grades rivals primarily based on adjustments they make to the problem surroundings. Challenges can process rivals with fixing a misconfiguration, mitigating a vulnerability, attacking a service, or another exercise the place success may be measured dynamically. When the grading system verifies adjustments to the surroundings state, it offers rivals with a hit token.

Problem Variation

Builders ought to embrace some stage of variation between completely different deployments of a problem to permit for various appropriate solutions to the identical problem. Doing so is essential for 2 causes. First, it helps promote a good competitors by discouraging rivals from sharing solutions. Second, it permits competitors organizers to reuse challenges with out shedding instructional worth. Challenges that may be accomplished quite a few instances with out leading to the identical reply allow rivals to be taught and hone their expertise by means of repeated apply of the identical problem.

Builders can introduce variation into challenges in a number of methods, relying on the kind of grading that they use:

  • Token-based variation. Challenges utilizing token-discovery or environment-verification grading can randomly generate distinctive tokens for every competitor when the problem is deployed. Builders can insert dynamically generated submission tokens into the problem surroundings (e.g., inserting guestinfo variables into VMs), they usually can copy them to the areas the place they count on rivals to obtain the problem solutions.
  • Query-and-answer variation. In question-and-answer challenges, builders can introduce variation by configuring completely different solutions to the identical questions or by asking completely different questions.

Problem Documentation

The 2 key paperwork builders should create in assist of their problem are the problem information and the answer information.

The problem information, which is seen to the rivals, offers a brief description of the problem, the abilities and duties the problem assesses, the situation and any background info that’s required to grasp the surroundings, machine credentials, and the submission space or areas.

The problem doc ought to describe the situation in a approach that rivals can simply observe and perceive. The problem situation and background info ought to keep away from logical leaps and the issue stage mustn’t hinge on info internationally ignored of the information.

The answer information offers a walk-through of 1 strategy to full the problem. Throughout testing, builders use the answer information to make sure the problem may be solved. Builders also can launch the answer information to the general public after the conclusion of the competitors to function a group studying useful resource.

The meant viewers for this information is the overall cybersecurity group. Consequently, builders ought to assume the reader is aware of primary IT and cybersecurity expertise, however is just not an professional within the subject. Screenshots and different pictures are useful additions to those guides.

Problem Testing and Assessment

After builders construct a problem, it ought to undergo a number of rounds of testing and overview. Builders take a look at challenges to make sure high quality, they usually overview them to estimate the problem’s issue.

Builders ought to carry out an preliminary spherical of testing to catch any errors that come up in the course of the problem deployment and initialization course of. They need to additionally make sure that rivals can absolutely remedy the problem in no less than a technique. A second spherical of testing needs to be performed by certified technical workers unfamiliar with the problem. Testers needs to be inspired to try fixing the problem on their very own however could also be supplied the developer’s resolution information for assist.

The testers ought to guarantee every problem meets the next high quality assurance standards:

  • The problem deploys as anticipated and with out errors.
  • The problem VMs are accessible.
  • The problem is solvable.
  • There are not any unintentional shortcuts to fixing the problem.
  • Problem directions and questions are correctly formatted and provides a transparent indication of what rivals should do.

Of their overview of the problem, testers ought to take notes in regards to the content material, together with estimates of issue and size of time it will take rivals to unravel. After testers full their overview, competitors organizers can study the issue assessments and evaluate every problem with others. This comparability ensures that simpler challenges stay in earlier rounds and are value fewer factors than challenges judged as tougher.

When deciding problem level allocations, organizers can use a base or normal rating allotment as a place to begin (e.g., all challenges are value 1,000 factors in the beginning of the method). Organizers can then enhance or lower level allocations primarily based on the out there issue information, protecting in thoughts that the principle purpose is for the variety of factors they allocate to a problem to instantly correspond with the trouble required for fixing it. Level allocations ought to contemplate each the issue and the time it takes to unravel the problem.

SEI Open Supply Purposes for Cybersecurity Problem Competitions

Builders can use a number of open supply purposes to develop challenges and to orchestrate cybersecurity competitions. The SEI has developed the next two purposes for operating cybersecurity competitions:

  • TopoMojo is an open supply lab builder and participant utility that builders can use to develop cybersecurity challenges. It offers digital workspaces during which problem growth can happen. The workspaces permit builders so as to add VMs, digital networks, and another sources which might be required for growing or fixing a single problem.
  • Gameboard is an open supply utility that organizers can use for orchestrating cybersecurity competitions. It permits organizers to create competitions that may both be crew or particular person primarily based and that include both single or a number of rounds. Challenges are organized into rounds and rivals try to unravel as many challenges as they’ll to maximise their rating. Gameboard makes use of the TopoMojo API to deploy the rivals’ sport house for every problem.

Gameboard additionally serves because the authoritative location for rivals to submit solutions or tokens. Furthermore, as a part of dealing with reply and token submissions, Gameboard has logging, brute drive protections, and different options to make sure the integrity of the competitors.

Determine 1 reveals how the TopoMojo and Gameboard purposes work together. Builders use TopoMojo workspaces to develop challenges. Opponents then use Gameboard to deploy and in- teract with challenges. When a participant deploys a problem, Gameboard will work together with the To- poMojo API to request a brand new sport house for the competitor. TopoMojo creates and returns the participant’s problem sport house.

Greatest Practices Assist Higher Cybersecurity Competitions

The event practices we’ve got highlighted on this publish are the results of the SEI’s expertise growing cybersecurity challenges for the President’s Cup Cybersecurity Competitors. Cybersecurity competitions present a enjoyable and fascinating strategy to train technical expertise, establish and acknowledge cybersecurity expertise, and interact college students and professionals within the subject. They will additionally function training and coaching alternatives. With america authorities, and the nation as a complete, going through a major scarcity within the cybersecurity workforce, cybersecurity competitions play an essential position in growing and increasing the workforce pipeline.

There isn’t a single strategy to run a contest, and there’s no one strategy to develop cybersecurity challenges. Nonetheless, these finest practices can assist builders make sure the challenges they create are efficient and interesting. Problem growth is the one most essential and time-consuming side of operating a cybersecurity competitors. It requires meticulous planning, technical growth, and a rigorous quality-assurance course of. In our expertise, these practices guarantee efficiently executed competitions and enduring, hands-on cybersecurity belongings that competitors organizers and others can reuse many instances over.

If you want to be taught extra in regards to the work we do to strengthen the cybersecurity workforce and the instruments we’ve got developed to assist this mission, contact us at

Leave a Reply

Your email address will not be published. Required fields are marked *