| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-25 | |||
| 10:01:11 | stephenfin | What's the risk? That it won't make RC1? | |
| 10:01:55 | gibi | that what we have today will be RC1 and it means we have ~0 days to let the rpc bump sit on master and get feedback if it breaks someting unexpected | |
| 10:02:16 | gibi | sure we have 2 weeks to provide RC2 | |
| 10:04:14 | gibi | I remember the accel_uuid RPC break that we discovered weeks later after merge | |
| 10:05:15 | gibi | but meh, whathever we merge today it will not have time to sit on master before it becomes stable/wallaby | |
| 10:05:53 | gibi | I did not see anything obviously wrong in the two extra patches but that is just me. | |
| 10:06:03 | stephenfin | I think this is an issue we've to deal with regardless of which patches go in | |
| 10:06:05 | bauzas | gibi: we can't merge it for RC2 | |
| 10:06:12 | gibi | bauzas: I know | |
| 10:06:20 | gibi | it is either today or Xena | |
| 10:06:25 | bauzas | yup | |
| 10:06:40 | bauzas | b/c we can't backport it down to stable/wallaby after today | |
| 10:07:03 | gibi | does Dan reviewed the two extra patches? | |
| 10:07:04 | bauzas | and this is not a regression | |
| 10:07:11 | bauzas | gibi: nope, I don't think | |
| 10:07:22 | stephenfin | He did yeah | |
| 10:07:30 | bauzas | we need to doublecheck the jobs we use | |
| 10:07:48 | stephenfin | he gave me feedback on fixing the client too | |
| 10:07:51 | bauzas | IIRC, grenade-multinode calls the resize API | |
| 10:08:09 | bauzas | stephenfin: again, you were fortunate we don't pass kwargs | |
| 10:08:18 | stephenfin | I know, but we don't so it's okay | |
| 10:08:37 | bauzas | because you're passing something named instance_type to something awaiting to get flavors | |
| 10:08:57 | bauzas | so maybe dansmith could have concerns with this | |
| 10:09:09 | stephenfin | the name of args doesn't matter | |
| 10:09:15 | stephenfin | it's only kwargs that matter | |
| 10:09:18 | stephenfin | standard Python | |
| 10:09:40 | stephenfin | I could call it my_terrible_terrible_awful_variable_name if I wanted to and things would be rost | |
| 10:09:42 | stephenfin | *rosy | |
| 10:09:46 | stephenfin | :) | |
| 10:10:09 | stephenfin | and the successful CI jobs prove as much | |
| 10:12:38 | stephenfin | bauzas: the grenade job runs resize tests https://zuul.opendev.org/t/openstack/build/07e5714177574a1fb513f4ae6b83bbc5/log/job-output.txt#4617 | |
| 10:12:53 | stephenfin | e.g. tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_resize_server_confirm | |
| 10:12:58 | bauzas | yup, was looking into it | |
| 10:13:39 | bauzas | I knew we had tempest tests but I was wondering whether we were calling them with the grenade multinode job | |
| 10:13:41 | gibi | I OK, so we have to move forward somehow. and I guess as a PTL I need to make the decision | |
| 10:13:54 | stephenfin | so we've got test coverage, the tests are passing and the code is sane (IMO at least) | |
| 10:15:03 | stephenfin | whatever risk exists with these changes also exists with the main change, so I don't see any reason to not include this if we're doing the RPC bump (which I do think we should, fwiw) | |
| 10:15:06 | stephenfin | gibi: pretty much | |
| 10:15:49 | gibi | I will take the risk of breaking things and I will use you both to fix any breakage if happens | |
| 10:15:58 | gibi | is that fair? | |
| 10:16:03 | stephenfin | wfm | |
| 10:16:31 | gibi | bauzas: ? | |
| 10:16:49 | bauzas | gibi: use your stick. | |
| 10:17:08 | bauzas | :p | |
| 10:17:17 | gibi | I have broom ;) | |
| 10:17:40 | bauzas | well, I don't ask you fifty nuances of -1s | |
| 10:18:10 | bauzas | ok, lemme squash the series then | |
| 10:18:17 | gibi | bauzas: OK | |
| 10:18:17 | bauzas | and hold your breathes | |
| 10:18:19 | gibi | I will +2it | |
| 10:18:35 | gibi | and we will hold up RC1 if the gate will need some recheck | |
| 10:23:36 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Bump the Compute RPC API to version 6.0 https://review.opendev.org/c/openstack/nova/+/761452 | |
| 10:23:55 | bauzas | gibi: dansmith: stephenfin: ^ squashed rebase | |
| 10:23:59 | bauzas | on your pencils, folks. | |
| 10:24:34 | stephenfin | Done | |
| 10:26:11 | gibi | on it | |
| 10:40:54 | gibi | bauzas: do we need to mention the RPC major bump in the releasenotes? | |
| 10:41:45 | stephenfin | I think bauzas is on dad duty | |
| 10:41:49 | stephenfin | probably wise | |
| 10:41:51 | gibi | or it is totally implemantation detail | |
| 10:42:03 | gibi | anyhow that can be done separately even on the stable/wallaby branch after the cut | |
| 10:42:48 | stephenfin | actually, the only release notes I can find are for when we drop RPC versions | |
| 10:42:55 | stephenfin | releasenotes/notes/conductor_rpcapi_v2_drop-9893c27bb32d9786.yaml | |
| 10:43:05 | stephenfin | so we can wait I guess | |
| 10:46:02 | gibi | yeah it expected to work transparently for the users and admins | |
| 10:46:06 | gibi | I geuss | |
| 10:47:03 | bauzas | gibi: actually, that's a good question | |
| 10:47:06 | gibi | stephenfin: could you check back to the prelude that is looks good to you? https://review.opendev.org/c/openstack/nova/+/782172 | |
| 10:47:15 | stephenfin | sure | |
| 10:47:18 | gibi | thanks | |
| 10:48:11 | stephenfin | yup, looks like my comments were addressed so lgtm | |
| 10:48:16 | gibi | thanks | |
| 11:22:27 | stephenfin | gibi: FYI https://storyboard.openstack.org/#!/story/2008750 | |
| 11:22:47 | stephenfin | I don't think that's a regression so it should block placement RC1 but we'll probably want to fix that soonish | |
| 11:22:51 | stephenfin | *shouldn't | |
| 11:29:02 | gibi | stephenfin: ack. ping me if I can help reviewing the fix | |
| 11:31:06 | gibi | lyarwood: when you are around there is a list of doc fixed piled up behind https://review.opendev.org/c/openstack/nova/+/781434 could you check back to that? | |
| 11:46:41 | openstackgerrit | Lee Yarwood proposed openstack/nova master: docs: Update the architecture diagram https://review.opendev.org/c/openstack/nova/+/782979 | |
| 11:46:41 | lyarwood | gibi: ack, will do now | |
| 11:58:06 | openstackgerrit | Vlad Gusev proposed openstack/nova stable/stein: Improve error log when snapshot fails https://review.opendev.org/c/openstack/nova/+/782962 | |
| 12:00:21 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/train: Improve error log when snapshot fails https://review.opendev.org/c/openstack/nova/+/782982 | |
| 12:00:50 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: Improve error log when snapshot fails https://review.opendev.org/c/openstack/nova/+/782962 | |
| 12:01:47 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/rocky: Improve error log when snapshot fails https://review.opendev.org/c/openstack/nova/+/782983 | |
| 12:02:17 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: Improve error log when snapshot fails https://review.opendev.org/c/openstack/nova/+/782984 | |
| 12:07:19 | openstackgerrit | Stephen Finucane proposed openstack/placement master: Correctly handle integrity errors on MySQL 8.x https://review.opendev.org/c/openstack/placement/+/782988 | |
| 12:18:00 | gibi | lyarwood: thanks | |
| 12:20:56 | noonedeadpunk | hey everyone! we see super weird issue with one of our patches where we update nova version to SHA 3de7fb7c327db348d04d15d4cd3c4f811a336126 and trying to perform V->X upgrade and run tempest. | |
| 12:21:18 | noonedeadpunk | So tempest fails with `Details: (TestServerBasicOps:test_server_basic_ops) Server 40ce5b33-d2d8-470e-b8c8-776c53efce8c failed to reach ACTIVE status and task state "None" within the required time (600 s). Current status: BUILD. Current task state: scheduling.` | |
| 12:21:51 | noonedeadpunk | No issues neither in compute nor in scheduler. And instance got rescheduled to compute, but never starts there | |
| 12:22:05 | noonedeadpunk | https://zuul.opendev.org/t/openstack/build/85c40976164c4b1785aefe0c1a4ddab9/log/logs/host/nova-compute.service.journal-11-24-25.log.txt#17356 | |
| 12:23:21 | noonedeadpunk | And SHA `5b65812ef2c6ac46bea30e2e603284eac232339e` was working properly (but yeah, that's an SHA before merging libvirt 6 requirement) | |
| 12:24:19 | noonedeadpunk | are we missing some config change here? | |
| 12:27:39 | lyarwood | V to W right? | |
| 12:27:48 | lyarwood | where can I find the tempest logs? | |
| 12:28:08 | lyarwood | If it cast from the conductor didn't make it to the compute then this would smell like a rabbitmq issue tbh | |
| 12:30:09 | jrosser | the tempest log is here https://zuul.opendev.org/t/openstack/build/85c40976164c4b1785aefe0c1a4ddab9/log/logs/openstack/aio1-utility/tempest_run.log.txt | |
| 12:36:50 | noonedeadpunk | lyarwood: it makes to the compute, since compute has `Instance 40ce5b33-d2d8-470e-b8c8-776c53efce8c has been scheduled to this compute host, the scheduler has made an allocation against this compute node but the instance has yet to start. Skipping heal of allocation: {'resources': {'DISK_GB': 1, 'MEMORY_MB': 256, 'VCPU': 1}}` | |
| 12:38:01 | noonedeadpunk | Except debug messages there's a single warning in the compute log, which is `This host appears to have multiple sockets per NUMA node. The `socket` PCI NUMA affinity will not be supported.` But I guess it should not be critical? | |
| 12:48:59 | noonedeadpunk | lyarwood: sorry, you was right, it's rabbit issue :( | |
| 12:49:16 | noonedeadpunk | found the reason here, thanks for the clue! | |