Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changes to schedule all reports, even for once #3840

Merged
merged 61 commits into from
Jan 20, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
b991534
Changes to schedule all reports
Rieven Nov 14, 2024
7351eda
remove form id
Rieven Nov 14, 2024
1af0832
Merge with main
Rieven Nov 14, 2024
3f5c77d
fix lang
Rieven Nov 14, 2024
e19b52e
Merge branch 'main' into feature/report_creations_are_scheduled
Rieven Nov 15, 2024
7adc85f
Update rocky/reports/templates/partials/export_report_settings.html
Rieven Nov 18, 2024
bb7ecb3
Update rocky/reports/templates/partials/export_report_settings.html
Rieven Nov 18, 2024
59384d6
Merge branch 'main' into feature/report_creations_are_scheduled
Rieven Nov 18, 2024
2507ef8
Merge branch 'main' into feature/report_creations_are_scheduled
Rieven Nov 20, 2024
f479f2f
set None atttrs
Rieven Nov 20, 2024
6bf723c
Merge branch 'main' of github.com:minvws/nl-kat-coordination into fea…
Rieven Nov 27, 2024
35e153d
Merge branch 'main' into feature/report_creations_are_scheduled
Rieven Nov 27, 2024
2761b30
fix lang
Rieven Nov 27, 2024
b53d3da
Merge branch 'feature/report_creations_are_scheduled' of github.com:m…
Rieven Nov 27, 2024
97248d7
Merge branch 'main' into feature/report_creations_are_scheduled
underdarknl Nov 27, 2024
2458f9f
fix for start dat adn time
Rieven Nov 27, 2024
aa4ab9e
Merge branch 'feature/report_creations_are_scheduled' of github.com:m…
Rieven Nov 27, 2024
ebed7c6
fixes for report schedules
Rieven Nov 27, 2024
025c358
fix lang and text
Rieven Nov 27, 2024
4d9a686
set back modal
Rieven Nov 27, 2024
f0a2459
fix datetime already datetime
Rieven Nov 27, 2024
1a28729
Merge branch 'main' into feature/report_creations_are_scheduled
Rieven Nov 27, 2024
25c74a0
Fix empty td
Rieven Nov 27, 2024
f06637b
Merge branch 'feature/report_creations_are_scheduled' of github.com:m…
Rieven Nov 27, 2024
6d143f8
Add additional check if task already run
jpbruinsslot Nov 27, 2024
b727ec6
Fix report types pils
Rieven Nov 28, 2024
4a98305
Merge branch 'fix/mula/schedule-report-once-check' of github.com:minv…
Rieven Nov 28, 2024
37355de
Merge branch 'main' into feature/report_creations_are_scheduled
Rieven Nov 28, 2024
eca64aa
Fix report names
Rieven Nov 28, 2024
4faa16b
Merge branch 'feature/report_creations_are_scheduled' of github.com:m…
Rieven Nov 28, 2024
a14d931
Merge branch 'main' of github.com:minvws/nl-kat-coordination into fea…
Rieven Nov 28, 2024
496414d
fix function input recipe
Rieven Nov 28, 2024
ca7c88a
fix test
Rieven Dec 2, 2024
d9e44e9
Merge branch 'main' into feature/report_creations_are_scheduled
Rieven Dec 2, 2024
79bb537
Merge branch 'main' into feature/report_creations_are_scheduled
Rieven Dec 2, 2024
ac6ad34
set back uuid string
Rieven Dec 2, 2024
3818812
Merge branch 'feature/report_creations_are_scheduled' of github.com:m…
Rieven Dec 2, 2024
4f8bceb
Merge branch 'main' into feature/report_creations_are_scheduled
Rieven Dec 3, 2024
023eb80
Merge branch 'main' into feature/report_creations_are_scheduled
Rieven Dec 3, 2024
3b1cfae
Merge branch 'main' into feature/report_creations_are_scheduled
Rieven Dec 3, 2024
e174b15
Merge branch 'main' into feature/report_creations_are_scheduled
Rieven Dec 3, 2024
bb235aa
Merge branch 'main' of github.com:minvws/nl-kat-coordination into fea…
Rieven Dec 11, 2024
073156b
fix id in worker for string
Rieven Dec 11, 2024
715d467
Merge branch 'main' into feature/report_creations_are_scheduled
Rieven Dec 12, 2024
f8e8e32
Merge branch 'main' of github.com:minvws/nl-kat-coordination into fea…
Rieven Dec 16, 2024
ad8980a
fix td
Rieven Dec 16, 2024
6dd9d1d
Merge branch 'main' into feature/report_creations_are_scheduled
ammar92 Dec 18, 2024
ac3a467
Merge branch 'main' of github.com:minvws/nl-kat-coordination into fea…
Rieven Jan 9, 2025
5b332b2
fix table field
Rieven Jan 9, 2025
8b012ce
fix time widget
Rieven Jan 9, 2025
74eab2b
fix onboarding also scheduled
Rieven Jan 9, 2025
889232e
Merge branch 'main' into feature/report_creations_are_scheduled
Rieven Jan 10, 2025
539be48
fix test
Rieven Jan 10, 2025
f65461a
Merge branch 'feature/report_creations_are_scheduled' of github.com:m…
Rieven Jan 10, 2025
b829402
Merge branch 'main' into feature/report_creations_are_scheduled
Rieven Jan 10, 2025
59714b0
add more test
Rieven Jan 10, 2025
4b74e70
Merge branch 'feature/report_creations_are_scheduled' of github.com:m…
Rieven Jan 10, 2025
c3d4dd8
Merge branch 'main' into feature/report_creations_are_scheduled
Rieven Jan 14, 2025
5b38f66
Merge branch 'main' into feature/report_creations_are_scheduled
Rieven Jan 15, 2025
6eebbff
add check for when scheduler is ready
Rieven Jan 16, 2025
4fa7934
fix test
Rieven Jan 16, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion octopoes/octopoes/models/ooi/reports.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,6 @@ class ReportRecipe(OOI):
parent_report_type: str | None = None
report_types: list[str]

cron_expression: str
cron_expression: str | None = None

_natural_key_attrs = ["recipe_id"]
14 changes: 8 additions & 6 deletions rocky/onboarding/templates/step_3c_setup_scan_ooi_detail.html
Original file line number Diff line number Diff line change
Expand Up @@ -63,12 +63,14 @@ <h3>{% translate "Creating an object" %}</h3>
</div>
</dl>
</div>
<div class="button-container">
<a href="{% url "step_report" organization.code %}?{{ request.GET.urlencode }}"
class="button">{% translate "Start scanning" %}</a>
<a href="{% url "complete_onboarding" organization.code %}"
class="button ghost">{% translate "Skip onboarding" %}</a>
</div>
<form method="post" class="inline">
{% csrf_token %}
<div class="button-container">
<button type="submit">{% translate "Start scanning" %}</button>
<a href="{% url "complete_onboarding" organization.code %}"
class="button ghost">{% translate "Skip onboarding" %}</a>
</div>
</form>
</section>
</main>
{% endblock content %}
74 changes: 49 additions & 25 deletions rocky/onboarding/views.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,12 @@
from katalogus.client import Plugin
from reports.report_types.definitions import ReportPlugins
from reports.report_types.dns_report.report import DNSReport
from reports.views.base import get_selection
from reports.views.generate_report import SaveGenerateReportMixin
from reports.views.base import BaseReportView, get_selection
from tools.models import GROUP_ADMIN, GROUP_CLIENT, GROUP_REDTEAM, Organization, OrganizationMember
from tools.ooi_helpers import get_or_create_ooi
from tools.view_helpers import Breadcrumb

from octopoes.models import OOI
from octopoes.models import OOI, Reference
from octopoes.models.ooi.dns.zone import Hostname
from octopoes.models.ooi.network import Network
from octopoes.models.ooi.web import URL
Expand All @@ -40,8 +39,10 @@
)
from rocky.exceptions import RockyError
from rocky.messaging import clearance_level_warning_dns_report
from rocky.scheduler import scheduler_client
from rocky.views.indemnification_add import IndemnificationAddView
from rocky.views.ooi_view import SingleOOIMixin, SingleOOITreeMixin
from rocky.views.scheduler import SchedulerView

User = get_user_model()

Expand Down Expand Up @@ -310,9 +311,11 @@ def get_context_data(self, **kwargs: Any) -> dict[str, Any]:

class OnboardingSetupScanOOIDetailView(
OrganizationPermissionRequiredMixin,
SingleOOITreeMixin,
IntroductionStepsMixin,
OnboardingBreadcrumbsMixin,
SingleOOIMixin,
BaseReportView,
SchedulerView,
TemplateView,
):
"""
Expand All @@ -322,6 +325,34 @@ class OnboardingSetupScanOOIDetailView(
template_name = "step_3c_setup_scan_ooi_detail.html"
current_step = 3
permission_required = "tools.can_scan_organization"
task_type = "report"

@staticmethod
def is_scheduler_enabled(organization: Organization) -> bool:
scheduler_id = f"report-{organization.code}"
return scheduler_client(organization.code).is_scheduler_ready(scheduler_id)

def post(self, request, *args, **kwargs):
parent_report_name_format, subreport_name_format = self.get_initial_report_names()
parent_report_type = self.get_parent_report_type()
report_recipe = self.create_report_recipe(
parent_report_name_format, subreport_name_format, parent_report_type, None
)
if self.is_scheduler_enabled(self.organization):
self.create_report_schedule(report_recipe, datetime.now(timezone.utc))

return redirect(
reverse("step_report", kwargs={"organization_code": self.organization.code})
+ get_selection(self.request, {"recipe_id": report_recipe.primary_key})
)

def get_ooi_pks(self) -> list[str]:
ooi = self.get_ooi(self.request.GET.get("ooi", ""))
hostname_ooi = [Hostname(name=ooi.web_url.tokenized["netloc"]["name"], network=ooi.network)]
return [hostname_ooi[0].primary_key]

def get_report_type_ids(self) -> list[str]:
return [self.request.GET.get("report_type", "")]

def get_context_data(self, **kwargs):
context = super().get_context_data(**kwargs)
Expand All @@ -330,7 +361,7 @@ def get_context_data(self, **kwargs):


class OnboardingReportView(
OrganizationPermissionRequiredMixin, SaveGenerateReportMixin, IntroductionStepsMixin, SingleOOIMixin, TemplateView
OrganizationPermissionRequiredMixin, IntroductionStepsMixin, SingleOOIMixin, BaseReportView, TemplateView
):
"""
10. The user already started the scan and is now waiting till scans are finished to generate the report.
Expand All @@ -341,32 +372,25 @@ class OnboardingReportView(
current_step = 4
permission_required = "tools.can_scan_organization"

def setup(self, request, *args, **kwargs):
super().setup(request, *args, **kwargs)
self.selected_oois = self.get_ooi_pks()
self.selected_report_types = self.get_report_type_ids()

def get_ooi_pks(self) -> list[str]:
ooi = self.get_ooi(self.request.GET.get("ooi", ""))
hostname_ooi = [Hostname(name=ooi.web_url.tokenized["netloc"]["name"], network=ooi.network)]
return [hostname_ooi[0].primary_key]

def get_report_type_ids(self) -> list[str]:
return [self.request.GET.get("report_type", "")]

def post(self, request, *args, **kwargs):
self.set_member_onboarded()

report_ooi = self.save_report([("Onboarding Report", "Onboarding Report")])
recipe_id = request.GET.get("recipe_id", "")

if report_ooi:
return redirect(
reverse("view_report", kwargs={"organization_code": self.organization.code})
+ "?"
+ urlencode({"report_id": report_ooi.reference})
if recipe_id:
reports = self.octopoes_api_connector.query(
"ReportRecipe.<report_recipe[is Report]",
valid_time=datetime.now(timezone.utc),
source=Reference.from_str(recipe_id),
)

return self.get(request, *args, **kwargs)
if reports:
return redirect(
reverse("view_report", kwargs={"organization_code": self.organization.code})
+ "?"
+ urlencode({"report_id": reports[0].reference})
)
return redirect(reverse("scheduled_reports", kwargs={"organization_code": self.organization.code}))

def set_member_onboarded(self):
member = OrganizationMember.objects.get(user=self.request.user, organization=self.organization)
Expand Down
26 changes: 8 additions & 18 deletions rocky/reports/forms.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ class ReportScheduleStartDateForm(BaseRockyForm):

start_time = forms.TimeField(
label=_("Start time (UTC)"),
widget=forms.TimeInput(format="%H:%M", attrs={"form": "generate_report"}),
widget=forms.TimeInput(format="%H:%M", attrs={"type": "time"}),
initial=lambda: datetime.now(tz=timezone.utc).time(),
required=True,
input_formats=["%H:%M"],
Expand All @@ -70,7 +70,13 @@ class ReportScheduleStartDateForm(BaseRockyForm):
label=_("Recurrence"),
required=True,
widget=forms.Select(attrs={"form": "generate_report"}),
choices=[("daily", _("Daily")), ("weekly", _("Weekly")), ("monthly", _("Monthly")), ("yearly", _("Yearly"))],
choices=[
("once", _("No recurrence, just once")),
("daily", _("Daily")),
("weekly", _("Weekly")),
("monthly", _("Monthly")),
("yearly", _("Yearly")),
],
)

def clean(self):
Expand Down Expand Up @@ -121,19 +127,3 @@ class CustomReportScheduleForm(BaseRockyForm):
end_date = forms.DateField(
label=_(""), widget=forms.HiddenInput(), initial=lambda: datetime.now(tz=timezone.utc).date(), required=False
)


class ParentReportNameForm(BaseRockyForm):
parent_report_name = forms.CharField(
label=_("Report name format"), required=False, initial="${report_type} for ${oois_count} objects"
)


class ChildReportNameForm(BaseRockyForm):
child_report_name = forms.CharField(
label=_("Subreports name format"), required=True, initial="${report_type} for ${ooi}"
)


class ReportNameForm(ParentReportNameForm, ChildReportNameForm):
pass
8 changes: 4 additions & 4 deletions rocky/reports/runner/report_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -101,9 +101,9 @@ def run(self, report_task: ReportTask) -> None:
for ooi in data:
ooi_human_readable = Reference.from_str(ooi).human_readable
subreport_name = now.strftime(
Template(recipe.subreport_name_format).safe_substitute(
ooi=ooi_human_readable, report_type=str(report_type.name)
)
Template(
"" if recipe.subreport_name_format is None else recipe.subreport_name_format
).safe_substitute(ooi=ooi_human_readable, report_type=str(report_type.name))
)
subreport_names.append((subreport_name, subreport_name))

Expand All @@ -112,7 +112,7 @@ def run(self, report_task: ReportTask) -> None:
)

if "${ooi}" in parent_report_name and oois_count == 1:
parent_report_name = Template(parent_report_name).safe_substitute(ooi=ooi[0].human_readable)
parent_report_name = Template(parent_report_name).safe_substitute(ooi=ooi_human_readable)

save_report_data(
self.bytes_client,
Expand Down
2 changes: 1 addition & 1 deletion rocky/reports/runner/worker.py
Original file line number Diff line number Diff line change
Expand Up @@ -243,7 +243,7 @@ def _start_working(
finally:
try:
# The docker runner could have handled this already
if scheduler.get_task_details(p_item.id).status == TaskStatus.RUNNING:
if scheduler.get_task_details(str(p_item.id)).status == TaskStatus.RUNNING:
scheduler.patch_task(p_item.id, status) # Note that implicitly, we have p_item.id == task_id
logger.info("Set status to %s in the scheduler for task[id=%s]", status, p_item.id)
except HTTPError:
Expand Down
73 changes: 39 additions & 34 deletions rocky/reports/templates/partials/export_report_settings.html
Original file line number Diff line number Diff line change
Expand Up @@ -15,53 +15,58 @@ <h2>{% translate "Report schedule" %}</h2>
or monthly. If you need the report just for a single occasion, select the one-time option.
{% endblocktranslate %}
</p>
<form class="inline" method="post" action="{{ current }}">
<form id="generate_report" class="inline" method="post" action="{{ next }}">
{% csrf_token %}
{% include "forms/report_form_fields.html" %}
{% include "partials/form/fieldset.html" with fields=report_schedule_form_recurrence_choice %}
{% include "partials/form/fieldset.html" with fields=report_schedule_form_start_date fieldset_parent_class="column-2" %}

{% if is_scheduled_report %}
<div>
<h2>{% translate "Report name" %}</h2>
<p>
{% blocktranslate trimmed %}
Please choose a start date, time and recurrence for scheduling your report(s).
If you select a date on the 28th-31st of the month, it will always be scheduled on the last day of the month.
Define a custom name format for your report(s). This format will be applied to all generated
(sub)reports.
{% endblocktranslate %}
</p>
<p>
{% blocktranslate trimmed %}
The date you select will be the reference date for the data set for your report.
Please allow for up to 24 hours for your report to be ready.
To make the report names more descriptive, you can include placeholders for the
object name, the report type and/or the reference date. For subreports and reports over a single object,
use the placeholder "${ooi}" for the object name, "${report_type}" for the report type and use a
<a href="https://strftime.org/" target="_blank" rel="noopener">Python strftime code</a> for the reference
date. For reports over multiple objects, use "${oois_count}" for the number of objects in the report.
{% endblocktranslate %}
</p>
<div class="column-2">
{% include "partials/form/fieldset.html" with fields=report_schedule_form_start_date %}

<p>
{% blocktranslate trimmed %}
For example, the format "${report_type} for ${ooi} at %x" could generate:
"DNS Report for example.com at 01/01/25".
{% endblocktranslate %}
</p>
</div>
<fieldset>
<div>
<label for="parent_report_name_format">{% translate "Report name format" %}</label>
<input type="text"
id="parent_report_name_format"
name="parent_report_name_format"
value="{{ initial_report_names.0 }}">
</div>
</fieldset>
{% if initial_report_names.1 %}
<fieldset>
<div>
<label for="subreport_name_format">{% translate "Subreports name format" %}</label>
<input type="text"
id="subreport_name_format"
name="subreport_name_format"
value="{{ initial_report_names.1 }}">
</div>
</fieldset>
{% endif %}
</form>
<form id="generate_report" class="inline" method="post" action="{{ next }}">
{% csrf_token %}
{% include "forms/report_form_fields.html" %}

{% if not is_scheduled_report %}
{% include "partials/report_names_header.html" %}
{% include "partials/report_names_form.html" %}

<button type="submit" form="generate_report">
{% translate "Generate report" %}<span class="icon ti-chevron-right" aria-hidden="true"></span>
</button>
{% else %}
{% include "partials/report_names_header.html" with recurrence=True %}
{% include "partials/form/fieldset.html" with fields=report_parent_name_form %}

{% if reports|length > 1 %}
{% include "partials/form/fieldset.html" with fields=report_child_name_form %}

{% endif %}
<button type="submit" form="generate_report">
{% translate "Generate report" %}<span class="icon ti-chevron-right" aria-hidden="true"></span>
</button>
{% endif %}
<button type="submit">
{% translate "Generate report" %}<span class="icon ti-chevron-right" aria-hidden="true"></span>
</button>
</form>
{% else %}
{% include "partials/return_button.html" with btn_text="Go back" %}
Expand Down
35 changes: 0 additions & 35 deletions rocky/reports/templates/partials/report_names_header.html

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -134,8 +134,8 @@
</td>
<td class="nowrap report_reference_date">{{ report.parent_report.observed_at|date }}</td>
<td class="nowrap report_creation_date">{{ report.parent_report.date_generated }}</td>
{% if report.children_reports %}
<td class="sticky-cell">
<td class="sticky-cell">
{% if report.children_reports %}
<button type="button"
class="expando-button icon ti-chevron-down"
data-icon-open-class="icon ti-chevron-down"
Expand All @@ -144,8 +144,8 @@
aria-expanded="false">
{% translate "Open children report object details" %}
</button>
</td>
{% endif %}
{% endif %}
</td>
</tr>
{% if report.children_reports %}
<tr class="expando-row">
Expand Down
Loading
Loading