| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-03-06 | |||
| 18:54:20 | mriedem | eandersson: ah..yeah we even have a troubleshooting entry for that in our docs from when oath upgraded to ocata | |
| 18:54:43 | mriedem | eandersson: https://docs.openstack.org/nova/latest/admin/support-compute.html#requested-microversions-are-ignored | |
| 18:54:55 | mriedem | eandersson: heh for that exactly same problem :) | |
| 18:55:04 | mriedem | *exact | |
| 18:55:08 | eandersson | It was just an odd one, as it worked before. =] | |
| 18:55:31 | eandersson | Plus does not help that we haven't had a weekend yet :p | |
| 18:55:40 | eandersson | Upgrading from Mitaka to Rocky... not easy lol | |
| 18:56:11 | melwitt | mriedem: don't have a great reason. at first, I had modeled it after the bdm uuid addition/migration, which used a separate class for the test cases. and I wasn't sure if it was sufficiently different from the rest of the tests and should be separated or not. I can lump it with the rest, no problem | |
| 18:56:14 | eandersson | Moving to Cells V2 has been super smooth so far thou | |
| 18:56:29 | mriedem | eandersson: that's good to hear | |
| 18:56:34 | mriedem | the cells thing that is | |
| 18:56:38 | eandersson | :p | |
| 18:56:50 | eandersson | We still have a lot of unaccounted performance issues that we are troubleshooting. | |
| 18:56:52 | mriedem | too bad you're late to submit a "upgrading from mitaka to rocky at blizzzzard" talk at the summit | |
| 18:56:57 | eandersson | haha | |
| 18:57:33 | mriedem | its not just cells v2 you're biting off, it's also placement | |
| 18:57:45 | mriedem | so you might be hitting some perf issues due to not scaling that out enough, | |
| 18:57:51 | mriedem | or nova-compute being way too chatty | |
| 18:58:03 | eandersson | It actually looks pretty good so far on the nova side, but something is talking to Neutron a lot more. | |
| 18:58:24 | dansmith | yeah that shouldn't be cells-related I don't think | |
| 18:58:30 | eandersson | Yea - don't think so either | |
| 18:58:33 | mriedem | are you doing a lot of live migrations from rocky? | |
| 18:58:35 | dansmith | eandersson: glad to hear things aren't terrible :) | |
| 18:58:49 | eandersson | We haven't dared to test live migration yet :p | |
| 18:59:13 | mriedem | ok because the only big new neutron-related thing in rocky i can think of is https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/neutron-new-port-binding-api.html | |
| 18:59:28 | dansmith | eandersson: are you trying to move to multiple cells or just upgrading to the point where it's required for a single one? | |
| 18:59:35 | eandersson | single one for now | |
| 18:59:41 | dansmith | gotcha | |
| 18:59:48 | eandersson | we want to move to multiple cells | |
| 18:59:52 | eandersson | but wasn't a priority | |
| 19:00:17 | dansmith | one thing at a time, for sure | |
| 19:00:29 | melwitt | mriedem: replied on the review. I think maybe I was thinking since it's special and temporary handling, maybe keep it separate for more obvious removal later. but none of that is strong reasoning. just wasn't sure what to do | |
| 19:01:51 | melwitt | eandersson: ++ also glad to hear things aren't terrible with cells :P | |
| 19:08:25 | mriedem | melwitt: there is something that needs fixing here https://review.openstack.org/#/c/638574/ | |
| 19:08:56 | melwitt | mriedem: ok, on it | |
| 19:09:45 | openstackgerrit | Dan Smith proposed openstack/nova master: Move cleaning conntrack to privsep. https://review.openstack.org/624594 | |
| 19:09:45 | openstackgerrit | Dan Smith proposed openstack/nova master: Move arping to privsep. https://review.openstack.org/624593 | |
| 19:09:46 | openstackgerrit | Dan Smith proposed openstack/nova master: Cleanup the exec_ebtables code a little. https://review.openstack.org/635434 | |
| 19:09:46 | openstackgerrit | Dan Smith proposed openstack/nova master: Move killing processes to privsep. https://review.openstack.org/635432 | |
| 19:09:47 | openstackgerrit | Dan Smith proposed openstack/nova master: Privsepify ipv4 forwarding enablement. https://review.openstack.org/635431 | |
| 19:09:47 | openstackgerrit | Dan Smith proposed openstack/nova master: Move slight bonkers IP management to privsep. https://review.openstack.org/624592 | |
| 19:09:48 | openstackgerrit | Dan Smith proposed openstack/nova master: Privsep the ebtables modification code. https://review.openstack.org/635435 | |
| 19:09:48 | openstackgerrit | Dan Smith proposed openstack/nova master: Move FP device creation and deletion to privsep. https://review.openstack.org/635433 | |
| 19:09:49 | openstackgerrit | Dan Smith proposed openstack/nova master: Move iptables rule fetching and setting to privsep. https://review.openstack.org/636508 | |
| 19:09:49 | openstackgerrit | Dan Smith proposed openstack/nova master: Move adding vlans to interfaces to privsep. https://review.openstack.org/635436 | |
| 19:09:50 | openstackgerrit | Dan Smith proposed openstack/nova master: Move router advertisement daemon restarts to privsep. https://review.openstack.org/639281 | |
| 19:09:50 | openstackgerrit | Dan Smith proposed openstack/nova master: Move dnsmasq restarts to privsep. https://review.openstack.org/639280 | |
| 19:09:51 | openstackgerrit | Dan Smith proposed openstack/nova master: Move setting of device trust to privsep. https://review.openstack.org/639283 | |
| 19:09:51 | openstackgerrit | Dan Smith proposed openstack/nova master: Move calls to ovs-vsctl to privsep. https://review.openstack.org/639282 | |
| 19:09:52 | openstackgerrit | Dan Smith proposed openstack/nova master: Cleanup the _execute shim in nova/network. https://review.openstack.org/639581 | |
| 19:09:52 | openstackgerrit | Dan Smith proposed openstack/nova master: Move final bridge commands to privsep. https://review.openstack.org/639580 | |
| 19:09:53 | openstackgerrit | Dan Smith proposed openstack/nova master: Cleanup no longer required filters and add a release note. https://review.openstack.org/639826 | |
| 19:09:53 | openstackgerrit | Dan Smith proposed openstack/nova master: We no longer need rootwrap. https://review.openstack.org/554438 | |
| 19:10:00 | mriedem | makes you feel like a BIG MAN | |
| 19:10:31 | sean-k-mooney | :) | |
| 19:11:05 | melwitt | lol. | |
| 19:12:07 | sean-k-mooney | efried: do you want the good news or the bad news | |
| 19:12:39 | mriedem | i got you a puppy on the way home from work, | |
| 19:12:39 | dansmith | I have some draft comments on one of melwitt's in that stack too, I need to go find and finish that | |
| 19:12:43 | mriedem | but then i got hungry and ate it | |
| 19:13:12 | dansmith | melwitt: okay bottom four of that privsep set are +W | |
| 19:13:23 | dansmith | assuming I didn't break anything.. the rebases were not bad, but not trivial | |
| 19:13:38 | melwitt | dansmith: cool, thanks for doing that | |
| 19:13:52 | melwitt | I made a comment on the old bottom patch to explain | |
| 19:13:52 | dansmith | aye | |
| 19:13:57 | dansmith | oh okay | |
| 19:14:07 | sean-k-mooney | efried: the good news is your change passed the fucntional job that runs the ironic api test and and python-ironclinet was installed from git | |
| 19:14:48 | sean-k-mooney | efried: the bad news is the tempest job that ran teh senario tests failed with this error http://logs.openstack.org/21/641421/2/check/nova-ironic-client-tempest/8faf490/controller/logs/screen-n-cpu.txt.gz?level=TRACE#_Mar_06_18_04_20_985307 | |
| 19:17:27 | sean-k-mooney | efried: anyway you should be able to use that to test your chagnes looks like your just need to set the ironic api version | |
| 19:24:09 | openstackgerrit | Chris Friesen proposed openstack/nova master: Add emulated TPM support to Nova https://review.openstack.org/631363 | |
| 19:24:09 | openstackgerrit | Chris Friesen proposed openstack/nova master: Split up _is_storage_shared_with() in libvirt driver https://review.openstack.org/639933 | |
| 19:24:29 | cfriesen | ignore these, debugging something that fails in zuul but works locally. ^ | |
| 19:26:32 | mriedem | melwitt: ok i think dansmith and i are both up through the instance mapping user_id data migration patch, so you can probably start addressing stuff and pushing | |
| 19:27:08 | melwitt | mriedem: ack, reading through | |
| 19:29:18 | cfriesen | mriedem: just wanted to say thanks for all the reviews in the past week. you're a machine. | |
| 19:31:53 | mriedem | cfriesen: yw | |
| 19:32:08 | mriedem | i'm also having a hard time not making an inappropriate machine-related comment here | |
| 19:32:12 | mriedem | feels like a trap | |
| 19:32:13 | dansmith | haha | |
| 19:32:17 | dansmith | I've been thinking | |
| 19:32:28 | dansmith | all I came up with that was appropriate was "that's why his hugs are so cold" | |
| 19:32:45 | mriedem | but i'd have to hug for anyone to know that | |
| 19:32:53 | dansmith | I can tell from a distance | |
| 19:34:01 | melwitt | pff lol | |
| 19:35:36 | mriedem | we'll also need to think about what we do about counting for instance mappings that have queued_for_delete=None (not yet migrated) | |
| 19:35:55 | mriedem | i think i would tend to count those since we do'nt know if the instance is deleted, so it's safer to be conservative | |
| 19:36:03 | melwitt | yeah, was just reading through that | |
| 19:36:08 | dansmith | I haven't gotten to any of the actual counting stuff | |
| 19:36:19 | dansmith | I figure best case, | |
| 19:36:28 | dansmith | we merge the data model and migration stuff | |
| 19:36:42 | dansmith | so we can get the things migrated in stein and then we're looking at train for the actual counting stuff | |
| 19:36:52 | dansmith | as I haven't looked over any of that other than to go answer one question for myself | |
| 19:37:00 | melwitt | +1 my initial thought is to be conservative and count the queued_for_delete=None ones | |
| 19:37:53 | mriedem | yeah i haven't gotten into the api changes that actually use the new counting stuff, and at this rate it's too late for me to be comfortable with that, | |
| 19:38:00 | mriedem | so i'd be happy to get the data model change and migration in stein | |
| 19:38:05 | mriedem | so there is less to migrate in train | |
| 19:38:32 | dansmith | yeah, having it migrated in stein before makes that stuff a lot better | |
| 19:38:39 | melwitt | I can't argue with that. it's a bummer, but I take responsibility for not having this all ready for review earlier in the cycle | |
| 19:39:36 | melwitt | I appreciate the review you're doing now and glad to be able to land some of it because of it | |
| 19:40:24 | melwitt | s/some of it/the migration stuff/ | |