Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-23
15:22:03 stephenfin I was afraid you'd say that :( Time to rework again
15:22:09 stephenfin Any idea where that old etherpad is?
15:22:10 mriedem btw this is the last time we did an rpc api major version bump https://review.opendev.org/#/c/541005/
15:22:31 mriedem https://etherpad.openstack.org/p/nova-network-removal-rocky
15:22:39 stephenfin I've a rough idea from converting all the API sample functional tests over but maybe I've missed some stuff
15:25:24 stephenfin gibi: Just in case, you haven't tried adding floating IP stuff to NeutronFixture or some subclass, have you?
15:25:52 gibi stephenfin: let me check some notification tests
15:26:27 stephenfin gibi: It would be local if so. We don't have it in tree
15:26:54 stephenfin at least searching for 'create_floatingip' doesn't turn up anything but mocks/the actual call
15:28:32 gibi stephenfin: unfortunately no, the IpPayload only contains fixed ips
15:28:44 stephenfin damn
15:28:46 stephenfin thanks for checking
15:28:49 gibi np
15:37:36 openstackgerrit Matt Riedemann proposed openstack/nova master: Make nova-next multinode and drop tempest-slow-py3 https://review.opendev.org/683988
15:37:37 mriedem efried: i think this is what you're looking for ^
15:37:50 efried thanks mriedem
15:48:22 dansmith mriedem: so on your remap sanity check patch,
15:48:33 dansmith mriedem: it looks to me like you're now letting whichever one comes last win, right?
15:49:10 dansmith mriedem: really what we want is to let the non-cell0 one win, if possible
15:49:49 mriedem dansmith: in the bury in cell0 case, if it's already mapped we don't bury it in cell0,
15:50:00 mriedem in the non-cell0 case, we log an error and map it to the cell we just scheduled it to
15:50:05 mriedem which would not be cell0
15:50:32 dansmith mriedem: oh I see, you created a helper but only call it in the non-cell0 case
15:50:33 mriedem i think those are both letting the non-cell0 ones win
15:50:37 mriedem yes
15:50:45 dansmith okay
15:50:45 dansmith I assumed you were calling it from both places
15:50:48 mriedem b/c i didn't want to have to mock 100 LOC just to test 1 line change in schedule_and_build_instances
15:50:59 dansmith sounds like cheating
15:51:39 mriedem schedule_and_build_instances is a monster
15:52:18 mriedem i also noted that if we drop that InstanceMappingNotFound pre-ocata compat in _bury_in_cell0 the _bury_in_cell0 method can just call the new _map_instance_to_cell method
15:52:44 mriedem we have a bunch of pre-ocata pre-cells v2 compat handlers all over the conductor task manager code which it'd be nice to remove
15:52:49 mriedem since they shouldn't be possible anymore
15:53:58 dansmith meh, I love piles of compat code
15:54:01 dansmith makes me feel nostalgic
15:54:24 mriedem all of the TODO(alaski)s to feel like an old sweater
15:54:25 mriedem i'll admit
15:54:32 mriedem s/to/do/
15:57:25 dansmith lol
16:03:06 stephenfin I'm almost certain we've discussed this before, but why do we drop support for old microversion in novaclient?
16:03:16 stephenfin Referring to the API part, rather than the CLI
16:03:24 stephenfin Doesn't osc use that?
16:07:28 mriedem gibi: a few comments in https://review.opendev.org/#/c/683947/ but not worth holding it up,
16:07:36 mriedem gibi: but we might want to reconsider the fault message that the user could see
16:08:01 mriedem "oh hi your cloud provider is doing an upgrade and you shouldn't be trying to resize your instance right now bob!"
16:08:43 mriedem stephenfin: where have we dropped support for old microversions in novaclient?
16:08:52 gibi mriedem: the funny thing is that I think we cannot hit that error after the fix
16:09:04 gibi mriedem: so we could even remove it
16:09:21 stephenfin mriedem: e.g. 01fb16533bf562f39fe822bc12b9cc34b8580359
16:09:30 mriedem gibi: well, technically your fix is in conductor checking it's config but the computes involved in the resize/cold migrate could have different config with different pins
16:09:53 mriedem stephenfin: that one broke osc
16:09:55 gibi mriedem: can we pin our computes differently? how will they talk to each other?
16:10:08 mriedem and they had to fix to avoid using novaclient and hit the api directly
16:10:26 mriedem gibi: well i meant if your conductor was unpinned but the computes were, or not restarted yet or something
16:10:28 mriedem after unpinning them
16:10:32 mriedem i'm nto saying that's normal
16:10:47 mriedem but i'd probably leave the error checking in place until we have a compute rpc api major version bump to 6.0
16:10:53 gibi mriedem: OK. I so it is a sort transinet. I can accept that
16:10:57 stephenfin Yup, I recall that coming up now. Any reason we opted to do that instead of reverting the changes to 'novaclient/v2/client.py'?
16:11:04 stephenfin that = call the API directly
16:11:22 mriedem gibi: so if you wanted, you could write a separate patch which (1) logs the upgrade stuff but doesn't include it in the PortUpdateFailed message, (2) adds a TODO to that PortUpdateFailed code to say we can drop it after compute RPC API is bumped to 6.0
16:11:50 gibi mriedem: sure. I can do that
16:11:51 mriedem stephenfin: it was released late in novaclient, end of the milestone before FF and i think osc found the problem too late
16:12:18 mriedem stephenfin: so a mixture of releasing breaking / major version things in novaclient too late in the release to catch them and (2) osc historically not doing functional testing
16:12:30 mriedem well, not doing good enough coverage with functional testing
16:13:40 mriedem and (3) me not realizing osc was going to be broken
16:14:06 stephenfin that makes sense
16:14:10 mriedem that was probably before i cared more about osc and just figured, "this has been deprecated in novaclient since newton, we can surely drop it and no one will care"
16:14:35 mriedem so having been burned before, and burning myself and others, that's why i'm less cavalier about just removing old stuff
16:15:40 stephenfin Yeah, certainly true for something like this
16:15:50 mriedem speaking of an upgrade-related burning sensation https://review.opendev.org/#/q/topic:bug/1843090+(status:open+OR+status:merged)
16:16:36 stephenfin I was holding off on that til some people had looked at the oslo.messaging bug
16:16:44 stephenfin though maybe that doesn't make sense
16:17:28 mriedem you mean gibi's RequestSpecImageSerializationFixture in the functoinal test?
16:17:33 mriedem it's an existing pattern / known issue
16:18:28 mriedem could also summon dansmith to review those since it's rpc pin related
16:18:33 mriedem and that's dan's middle name
16:18:50 mriedem dan rpc-pin-and-sometimes-evacuate smith
16:19:15 stephenfin If dansmith doesn't get to it by tomorrow, I will
16:29:48 mriedem gibi: i think we could drop the new and redundant unit test in https://review.opendev.org/#/c/683948/
16:29:52 mriedem the functional test covers it
17:24:01 openstackgerrit Merged openstack/nova master: Add note about needing noVNC >= v1.1.0 with using ESX https://review.opendev.org/682946
17:30:21 bbobrov_ hi! In https://opendev.org/openstack/nova/src/branch/master/nova/servicegroup/drivers/db.py#L99 messaging.MessagingTimeout is getting caught. My understanding is that service_ref.save() only interacts with the database. How can MessagingTimeout happen?
17:38:48 sean-k-mooney bbobrov: if this is executing on the compute node then the save call would invoke the db update via rpc as the comptue services do not have database access
17:41:09 sean-k-mooney bbobrov: so on the concoctor,api and schduler this shoudl not raise a messaging timeout but it can in the compute agents
17:41:32 bbobrov sean-k-mooney: understood, thanks
17:45:55 openstackgerrit Matt Riedemann proposed openstack/nova master: Rename Claims resources to compute_node https://review.opendev.org/679470
17:50:20 openstackgerrit Matt Riedemann proposed openstack/nova master: Add a prelude for the Train release https://review.opendev.org/683327
17:53:47 bbobrov #2 then. there is a `periodic_enable` option, and in nova codebase it seems to affect only https://opendev.org/openstack/nova/src/branch/master/nova/service.py#L205 . if true, self.periodic_tasks is scheduled to be run by add_dynamic_timer
17:54:08 bbobrov But there is no implementation of periodic_tasks or run_periodic_tasks anywhere in Nova. The method in the base scheduler class does `pass`. Are there any implementation of run_periodic_tasks in Nova that i miss or they can come from somewhere else?
17:56:58 sean-k-mooney we use the one form oslo service https://github.com/openstack/oslo.service/blob/master/oslo_service/periodic_task.py
18:07:56 mriedem melwitt: do you know if tripleo has any kind of tooling that pokes a cell mq/db on upgrade to make sure it's ok before saying everything is good to go in the api?
18:22:38 bbobrov sean-k-mooney: thanks, it makes sense now
18:32:57 lyarwood mriedem: we are only just introducing full support for multi cell deployments now in 16 but I can ask around about that in the morning.
18:33:48 lyarwood owalsh: ^ unless you're around and know?
18:34:52 mriedem lyarwood: yeah in grenade upstream jobs we don't have multiple cells, just cell0 and cell1
18:34:57 mriedem trying to debug https://bugs.launchpad.net/nova/+bug/1844929
18:34:58 openstack Launchpad bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed]
18:35:35 mriedem it looks like we upgrade code to train, restart scheduler, it starts up, even hits the cell1 database to pull compute nodes and instances for it's in-memory cache, and then on the first scheduling attempt after that we hang
18:37:45 dansmith mriedem: scheduler doesn't need rabbit to do that, so the rabbit errors in your bug report would be unrelated I'd think

Earlier   Later