Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-25
15:28:40 stephenfin Or maybe not
15:29:05 sean-k-mooney i dont know i was eating soup when i taught of it so havent checked.
15:29:13 sean-k-mooney ill be back in 5
15:29:26 stephenfin Ha!
15:29:28 stephenfin OK :)
15:31:42 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Remove usage of migrateToURI{2} APIs https://review.openstack.org/567258
15:31:55 sean-k-mooney ok back :) so ya post = {'rebuild': {'imageRef': server['image']['id']}} same image
15:32:37 kashyap cfriesen_: When you can, a fresh pair of eyes are appreciated; spent too much time fiddling with the above today while resolving the rebase conflict.
15:34:25 stephenfin sean-k-mooney: I think I did that so I didn't have to do more mocking of the image service. I'm surprised the check is that "clever" though. I was expecting it to simply check if imageRef or something was set
15:36:50 sean-k-mooney stephenfin: it might not be the issue but its somthing im going to look into. its the best lead i have so far
15:57:51 melwitt sean-k-mooney: I proposed https://review.openstack.org/585530 for the noop plugin thing
15:58:56 melwitt stephenfin ^
15:59:26 sean-k-mooney melwitt: are you sure that is the issue. i was not seeing that issue in the functional test i created
15:59:44 melwitt pretty sure based on past commits that added the previous plugins
15:59:55 melwitt but I didn't know how to test it
16:00:06 sean-k-mooney did you see https://review.openstack.org/#/c/585693/
16:00:13 melwitt yeah, just opened it
16:00:29 sean-k-mooney i was not able to reproduce the error but i was able to load all the plugins correctly without the error
16:00:48 melwitt yeah. it's something to do with the packaging install I think
16:01:14 sean-k-mooney as in when you pip install?
16:01:27 melwitt yeah, is my guess
16:01:43 stephenfin melwitt, sean-k-mooney: probably this https://hynek.me/articles/testing-packaging/
16:01:46 melwitt when the setup.cfg is processed
16:02:55 stephenfin tl;dr: What melwitt said
16:03:37 sean-k-mooney stephenfin: ok how do we create a test for this to ensure we dont miss it in the future
16:03:55 sean-k-mooney stephenfin: or are you suggesting adding a src dir?
16:04:17 stephenfin sean-k-mooney: That I'm not sure about. No one else puts stuff inside a src dir so we can't really do that either
16:04:27 stephenfin sean-k-mooney: Assume it's a once off move on?
16:04:31 stephenfin *and move on
16:05:09 sean-k-mooney stephenfin: i guess we can add a test to one of the ci jobs.
16:06:07 sean-k-mooney ill add it to my list but let go with melwitt fix for now then.
16:06:25 melwitt we could make a dumb test that walks the entries in setup.cfg and raises if something is missing
16:06:56 melwitt I don't know of a smarter way to do it off the top of my head
16:07:46 sean-k-mooney melwitt: that is what i was thinking ya. specifically parsting the os_vif entrypoint and checking the plugin class path are rooted in a dir listed in packages
16:08:41 stephenfin You could do fancy things with pkg_resources too. I really don't think it's worth the effort though. If it bites us again, maybe then
16:10:03 sean-k-mooney i dont like leaveing gotchas like that lieing around. its not super high on my list but its now on my list
16:13:25 jangutter wow. that is doubleplusspam.
16:13:34 sean-k-mooney yeah spam
16:21:04 openstackgerrit Eric Fried proposed openstack/nova master: [placement] Add /reshaper handler for POST https://review.openstack.org/576927
16:21:05 openstackgerrit Eric Fried proposed openstack/nova master: reshaper: Look up provider if not in inventories https://review.openstack.org/585033
16:21:06 openstackgerrit Eric Fried proposed openstack/nova master: Make get_allocations_for_resource_provider sane https://review.openstack.org/584598
16:21:07 openstackgerrit Eric Fried proposed openstack/nova master: Report client: Real get_allocs_for_consumer https://review.openstack.org/584599
16:21:08 openstackgerrit Eric Fried proposed openstack/nova master: Report client: get_allocations_for_provider_tree https://review.openstack.org/584648
16:21:09 openstackgerrit Eric Fried proposed openstack/nova master: Report client: _reshape helper, placement min bump https://review.openstack.org/585034
16:21:10 openstackgerrit Eric Fried proposed openstack/nova master: WIP: Report client: update_from_provider_tree w/reshape https://review.openstack.org/585049
16:21:11 openstackgerrit Eric Fried proposed openstack/nova master: WIP: Compute: Handle reshaped provider trees https://review.openstack.org/576236
16:36:16 sean-k-mooney stephenfin: its passing the schduler because the network_metadata only contains info for the first instance on rebuild
16:36:43 stephenfin what do you mean?
16:37:25 sean-k-mooney stephenfin: on rebuild the network_metadata in the numa topology filter only list 1 of the 2 physnets requested
16:37:37 sean-k-mooney stephenfin: the one form the attached interface is missing
16:39:57 stephenfin sean-k-mooney: Yeah, I'm seeing something possibly related here. In my case, the info_cache.network_info is empty when I attempt to populate the request spec
16:40:11 stephenfin I'm trying to figure out what populates that in he first place in case we have that mocked out too
16:41:49 sean-k-mooney stephenfin: this https://review.openstack.org/#/c/585339/5/nova/objects/request_spec.py
16:42:59 sean-k-mooney i wonder if i need to call that again on attach.
16:43:56 sean-k-mooney there getting quicker at killing the spam
16:44:05 stephenfin sean-k-mooney: It should be called here https://review.openstack.org/#/c/585339/5/nova/conductor/manager.py
16:45:07 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: Add functional tests for cold migrate with NUMA networks https://review.openstack.org/585819
16:45:38 stephenfin sean-k-mooney: But in the test above, it seems instance.info_cache.network_info is unpopulated. I don't know why
16:45:41 sean-k-mooney ok ill see if that does the right thing. that code is being called
16:48:58 sean-k-mooney so ya the instance.info_cache.network_info only has 1 vif so it was not updated when we did the attach
16:49:37 stephenfin sean-k-mooney: Yup, likely the same thing I'm seeing so. I wonder what causes that to be refreshed?
16:50:07 stephenfin I guess we need to call ComputeManager._heal_instance_info_cache
16:51:10 sean-k-mooney its proably missing in the fake driver
16:55:41 sean-k-mooney stephenfin: looks like that is normally called by neutron when it updates the port
17:01:25 stephenfin sean-k-mooney: I can only see it when we attach a fixed IP, no?
17:02:10 sean-k-mooney i dont see any calls to it directly in the codebase. its spawned as a periodic task so not sure how to call it
17:02:31 sean-k-mooney i just ran wait a breakpoint and it was never called during the test run
17:06:55 stephenfin sean-k-mooney: Yeah, and it is getting set in a production system. Dodgy mocks again
17:07:52 sean-k-mooney ya any idea of how to run them?
17:08:14 sean-k-mooney if i create an instance of the compute manager can i just call it?
17:09:13 stephenfin Not a clue. I guess I need to figure out how to set a watch on an attribute of an o.v.o object so I can see the callers?
17:09:46 openstackgerrit melanie witt proposed openstack/nova stable/ocata: [stable only] Always commit quota decrement if build request found https://review.openstack.org/582413
17:11:59 xgerman_ hi, I am from Octavia and have a question: I had a compute host shutdown. Issued a nova delete and was checking in neutron if a port got detached… but that didn’t happen for 5 minutes. Wonder if there is a nova setting or if I need to increase that timeout…
17:12:09 stephenfin sean-k-mooney: fwiw, I know it works in "production". By default, DevStack deploys with an flat (l2) "public" network and vxlan "private" network. If I configure the two of those to use different NUMA nodes, boot the instance with a port from public, add a port from private and rebuild, it fails with NoValidHost
17:12:28 stephenfin So this has to be a mock not doing something but I just don't know what :)
17:13:30 sean-k-mooney stephenfin: my guess is that the comptue service we start is not running periodic tasks or the neutron mock is not sending the correct event
17:14:08 sean-k-mooney on the plus side i think i figured out how to call the heal method.
17:14:38 stephenfin I don't think it's the periodic task. If I put a breakpoint there (well, LOG.warning), the info_cache.network_info attribute looks correct beforehand
17:15:34 stephenfin sean-k-mooney: This looks promising https://github.com/openstack/nova/blob/master/nova/network/base_api.py#L33-L59
17:24:05 sean-k-mooney stephenfin: we could just add a floating ip to the instance i guess.
17:24:20 stephenfin sean-k-mooney: Exactly what I'm trying to do at the moment
17:24:30 stephenfin well, fixed
17:24:58 sean-k-mooney do you know what context object we need
17:25:13 stephenfin In what context?
17:25:58 sean-k-mooney well we have a reference to the compute instance and call the heal fuction but i dont have a context to pass it
17:26:08 sean-k-mooney same with those api methods
17:26:26 stephenfin Ah, for those, I've no idea :(
17:26:47 stephenfin Unless you do it from the NeutronFixture, in which case I think self.context is a thing?
17:26:59 stephenfin We obviously shouldn't be doing that in the functional test itself
17:27:32 sean-k-mooney looks like we can call get_instance_nw_info and it will update the cache
17:29:49 openstackgerrit melanie witt proposed openstack/nova stable/ocata: [stable only] Always commit quota decrement if build request found https://review.openstack.org/582413
17:42:57 jangutter Wow, looks like the killbot is getting quicker!
17:52:06 sean-k-mooney stephenfin: the periodic tasks are disable on the compute manager that is created by self._test_create_server_with_networks
17:54:17 openstackgerrit Merged openstack/os-vif master: Add vif_plug_noop to setup.cfg packages https://review.openstack.org/585530
18:02:02 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: Add additional functional tests for NUMA networks https://review.openstack.org/585385
18:17:47 stephenfin sean-k-mooney, dansmith, mriedem_away: A'ight, I've got a bus to Dublin in 6 hours and still need to pack. Gotta wrap this up :(
18:19:03 dansmith stephenfin: does that test work?
18:19:44 stephenfin dansmith: Nope. I think sean-k-mooney and I have the tests mostly right. Remaining blocker is that instance.info_cache.network_info isn't being updated. That happens when running IRL so I suspect a mocking issue but I've at a loss to figure out what I'm missing
18:20:11 dansmith okay that's the part I'm most suspicious about
18:20:13 dansmith you're saying if you're running devstack or something it works?

Earlier   Later