| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-09-01 | |||
| 06:33:57 | opendevreview | Amit Uniyal proposed openstack/nova stable/2023.1: Refactor CinderFixture https://review.opendev.org/c/openstack/nova/+/893464 | |
| 06:33:58 | opendevreview | Amit Uniyal proposed openstack/nova stable/2023.1: Add function to get all attachments in Cinder.API module https://review.opendev.org/c/openstack/nova/+/893465 | |
| 06:33:59 | opendevreview | Amit Uniyal proposed openstack/nova stable/2023.1: Reproducer for dangling bdms https://review.opendev.org/c/openstack/nova/+/893466 | |
| 06:35:30 | opendevreview | Amit Uniyal proposed openstack/nova stable/2023.1: Delete dangling bdms https://review.opendev.org/c/openstack/nova/+/893467 | |
| 06:47:10 | opendevreview | Amit Uniyal proposed openstack/nova stable/zed: Adds regression functional test for 1980720 https://review.opendev.org/c/openstack/nova/+/893468 | |
| 06:47:11 | opendevreview | Amit Uniyal proposed openstack/nova stable/zed: Refactor CinderFixture https://review.opendev.org/c/openstack/nova/+/893469 | |
| 06:47:12 | opendevreview | Amit Uniyal proposed openstack/nova stable/zed: Add function to get all attachments in Cinder.API module https://review.opendev.org/c/openstack/nova/+/893470 | |
| 06:47:13 | opendevreview | Amit Uniyal proposed openstack/nova stable/zed: Reproducer for dangling bdms https://review.opendev.org/c/openstack/nova/+/893471 | |
| 06:47:14 | opendevreview | Amit Uniyal proposed openstack/nova stable/zed: Delete dangling bdms https://review.opendev.org/c/openstack/nova/+/893472 | |
| 07:27:41 | opendevreview | Amit Uniyal proposed openstack/nova stable/yoga: Adds regression functional test for 1980720 https://review.opendev.org/c/openstack/nova/+/893474 | |
| 07:27:42 | opendevreview | Amit Uniyal proposed openstack/nova stable/yoga: Refactor CinderFixture https://review.opendev.org/c/openstack/nova/+/893475 | |
| 07:27:43 | opendevreview | Amit Uniyal proposed openstack/nova stable/yoga: Add function to get all attachments in Cinder.API module https://review.opendev.org/c/openstack/nova/+/893476 | |
| 07:27:44 | opendevreview | Amit Uniyal proposed openstack/nova stable/yoga: Reproducer for dangling bdms https://review.opendev.org/c/openstack/nova/+/893477 | |
| 07:27:45 | opendevreview | Amit Uniyal proposed openstack/nova stable/yoga: Delete dangling bdms https://review.opendev.org/c/openstack/nova/+/893478 | |
| 07:40:52 | bauzas | morning folks | |
| 07:52:47 | bauzas | melwitt: on it | |
| 07:53:02 | bauzas | thanks for the quick follow-ups | |
| 07:53:47 | bauzas | auniyal: your series was tagged as a blueprint | |
| 07:54:17 | bauzas | auniyal: accordingly, you can't backport it in stable branches as it would change a behavior | |
| 07:57:25 | bauzas | see the backport policy rules that are given here https://docs.openstack.org/project-team-guide/stable-branches.html#appropriate-fixes | |
| 08:07:08 | ykarel | Hi is there some known issue with test_reboot_server_hard, recently saw many failures | |
| 08:07:45 | ykarel | Fails like AssertionError: time.struct_time(tm_year=2023, tm_mon=9, tm_mday=1, tm_hour=7, tm_min=26, tm_sec=24, tm_wday=4, tm_yday=244, tm_isdst=0) not greater than time.struct_time(tm_year=2023, tm_mon=9, tm_mday=1, tm_hour=7, tm_min=26, tm_sec=24, tm_wday=4, tm_yday=244, tm_isdst=0) | |
| 08:14:39 | ykarel | k didn't find a bug report, so reported https://bugs.launchpad.net/tempest/+bug/2033752 | |
| 08:14:45 | ykarel | it's started just few hours back | |
| 08:41:58 | bauzas | ykarel: oh | |
| 08:42:08 | bauzas | probably related to some recent merge we did | |
| 08:42:47 | bauzas | because now we call Cinder when rebooting so the performance could be impactedcx | |
| 08:53:01 | frickler | elodilles: https://review.opendev.org/c/openstack/nova/+/871968 is still blocked by your W-1, can you please check? | |
| 08:54:29 | frickler | bauzas: calling cinder for a reboot? I'm curious about that, do you happen to have a reference? | |
| 08:55:08 | frickler | ah, https://review.opendev.org/c/openstack/nova/+/882284 ? | |
| 08:55:12 | bauzas | frickler: see the bug report | |
| 08:55:16 | bauzas | yea | |
| 08:56:44 | elodilles | frickler: looking, thanks | |
| 08:57:03 | bauzas | sean-k-mooney: dansmith: auniyal: looks like https://review.opendev.org/c/openstack/nova/+/882284 created a gate issue | |
| 08:57:04 | frickler | a reno referencing "this change" is weird wording btw | |
| 08:57:44 | bauzas | frickler: we can modify this | |
| 08:57:53 | auniyal | bauzas, ack | |
| 08:58:04 | bauzas | in general we say "Now, Nova does X or Y" | |
| 08:58:26 | bauzas | or "With this new feature named "blah", now we have" | |
| 09:02:15 | bauzas | auniyal: I'm working on a patch that will conditionnally check the volumes if only BDMs exist | |
| 09:02:51 | auniyal | ack bauzas, any thing I can help with | |
| 09:03:43 | bauzas | we may need core reviewers | |
| 09:04:36 | frickler | elodilles: thx, now just someone needs to retrigger the W+1 I think | |
| 09:09:03 | elodilles | frickler: i did it just before you wrote :) | |
| 09:10:09 | bauzas | auniyal: actually, we can't avoid a Cinder API call for getting the volumes | |
| 09:10:53 | bauzas | volume attachments* | |
| 09:11:02 | auniyal | yeah, I was about to say | |
| 09:11:11 | auniyal | saw the comment on bug report | |
| 09:11:56 | auniyal | cinder calls are separate from bdms what nova have | |
| 09:12:25 | auniyal | can we increase reboot time check ? | |
| 09:14:41 | bauzas | oh wait | |
| 09:15:09 | bauzas | this isn't calculating the time it boots | |
| 09:15:15 | bauzas | this is just getting the uptime | |
| 09:19:50 | SvenKieske | systemd-analyze time gives you boot times :) | |
| 09:20:04 | SvenKieske | without any context | |
| 09:23:38 | bauzas | that's weird then | |
| 09:24:04 | bauzas | https://github.com/openstack/tempest/blob/28ba6ee268bdf5d18a13a0dc7c7fea2c3a3dfcce/tempest/common/utils/linux/remote_client.py#L84-L88 is just giving me when the server was booted | |
| 09:24:56 | bauzas | and https://github.com/openstack/tempest/blob/master/tempest/api/compute/servers/test_server_actions.py#L127-L128 is just comparing the timestamps to check that the VM was actually rebooted | |
| 09:25:11 | bauzas | this isn't at all a performance check | |
| 09:30:55 | SvenKieske | I'm not understanding your last sentence: do you mean this isn't supposed to be a performance check, or are you indicating that this should be a performance check but fails to do so? | |
| 09:31:52 | SvenKieske | depending on what stage the virtual machine is in, the time comparison could be flaky, during early boot especially, if time is not correctly synced. | |
| 09:33:18 | SvenKieske | I'd also strongly advise not to directly get stuff from proc-fs. why not use a native python function instead at least? | |
| 09:33:21 | bauzas | SvenKieske: Tempest gets the uptime, subs it from the current time so it gets the local time when the server was booted, then reboots the server and does the same math with uptime | |
| 09:34:07 | bauzas | and then there is an assertion check that verifies that the reboot timestamp is greater than the original timestamp (meaning the OS was actually rebooted) | |
| 09:34:40 | bauzas | https://github.com/openstack/tempest/blob/28ba6ee268bdf5d18a13a0dc7c7fea2c3a3dfcce/tempest/api/compute/servers/test_server_actions.py#L127-L128 and https://github.com/openstack/tempest/blob/28ba6ee268bdf5d18a13a0dc7c7fea2c3a3dfcce/tempest/api/compute/servers/test_server_actions.py#L109 | |
| 09:34:53 | SvenKieske | I hope that test never runs during dailight saving time adjustments *just kidding* | |
| 09:35:16 | bauzas | UTC exists for a good reason :) | |
| 09:35:52 | bauzas | basically, this failing check means that the server *wasn't* rebooted | |
| 09:35:54 | SvenKieske | I'm strongly reminded about this now, don't know why: https://gist.github.com/timvisee/fcda9bbdff88d45cc9061606b4b923ca | |
| 09:36:11 | SvenKieske | this is really a bad check, to be blunt | |
| 09:36:48 | SvenKieske | just use systemd for this stuff already guys, it's there, it works, I know it has it's own bugs, but it's better then the stuff that was before it. | |
| 09:37:09 | auniyal | in test reboot_time should be more then (first) boot_time --- but its failing because it same. | |
| 09:37:09 | auniyal | https://paste.opendev.org/show/bk8aK3ZbjpeDYs6swcoQ/ | |
| 09:37:14 | SvenKieske | e.g. you can easily check the boot counter with systemd | |
| 09:37:30 | bauzas | auniyal: which means that the server *wasn't* rebooted | |
| 09:37:41 | auniyal | yeah | |
| 09:37:56 | bauzas | I'm just checking the logs | |
| 09:38:07 | bauzas | we may have had an exception in the reboot | |
| 09:39:06 | bauzas | oh wait | |
| 09:39:15 | SvenKieske | well that could also be true :) but just to check the wall clock seems not really reliable in the long way, I bet you get false positives someday (when clocks will be out of sync in VMs, and they will be, if the ntp setup is not bulletproof) | |
| 09:39:23 | bauzas | if we ssh the guest too soon, the reboot may not be done yet | |
| 09:39:45 | bauzas | SvenKieske: I don't think it's an instrumentation issue | |
| 09:39:52 | SvenKieske | so you need to detect the reboot, before ssh'ing in order to detect the reboot ;) | |
| 09:40:05 | auniyal | but still will it saw same time ? time must have updated right ? | |
| 09:40:18 | bauzas | auniyal: not if the guest wasn't rebooted *yet* | |
| 09:40:36 | bauzas | yay, another waiter ! | |
| 09:42:43 | auniyal | yeah, so it took boot_time, reboot (and waited until get ACTIVE), then took new_boot_time, must be greater, | |
| 09:43:04 | auniyal | or we should set our own clock counter in test ?? | |
| 09:44:15 | auniyal | and verify if we are able to run any cmd at all in system after reboot ? | |
| 09:45:33 | SvenKieske | well you're already doing that if you read the system time, you "just" need to verify that this reading was successfull and not NULL :) | |
| 09:46:10 | auniyal | yeah, that should be enough to say server rebooted | |
| 09:54:59 | bauzas | okay, I think I spotted | |
| 09:55:13 | bauzas | we reverted the reboot due to us unable to reach cinder | |
| 10:00:47 | auniyal | bauzas https://zuul.opendev.org/t/openstack/build/0bf10393fa25419c8f3a072ab091f0da/log/controller/logs/screen-n-cpu.txt#12033 this right ? | |
| 10:01:43 | bauzas | yes | |
| 10:02:36 | bauzas | the keystone service catalog isn't containing the volumev3 endpoint url, that's what's saying the exception | |
| 10:04:54 | bauzas | but heh, https://1e11be38b60141dbb290-777f110ca49a5cd01022e1e8aeff1ed5.ssl.cf1.rackcdn.com/893401/5/check/neutron-ovn-tempest-ovs-release/f379752/controller/logs/local_conf.txt | |
| 10:04:59 | bauzas | "disable_service cinder" | |
| 10:05:10 | auniyal | okay, its reverted but still shouldn't /proc/uptime get updated ? | |
| 10:07:39 | auniyal | still great test though it caught it or navigated us to the reason | |