Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-02-18
12:16:40 sean-k-mooney ther is only 1 rp
12:17:08 sean-k-mooney well there are 3 1 for each compute and then 1 for the fake device
12:17:17 sean-k-mooney so the fake driver is incomplete
12:18:39 sean-k-mooney i also noticed the devstack plugin is incorrect. it does not properly support deploy just the comptue agent like it sugges. it will not create the correct cybog config so the agent will be installed but wont be able to connect to rabbitmq
12:19:30 sean-k-mooney dansmith: so based on the fact that the RP is hardcoded im going to say creating a tempory multi node job will be more work then i had guessed
12:19:57 sean-k-mooney it could still be done but i would have to modify the cyborg code to fix the devstack plugin and correct the hardcoding
12:24:33 gibi sean-k-mooney: ack thanks, I will get back to that patch after I finished with the whole sereies
12:24:36 gibi series
12:34:17 openstackgerrit Lee Yarwood proposed openstack/nova master: virt: Provide block_device_info during rescue https://review.opendev.org/700811
12:34:17 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Add support for stable device rescue https://review.opendev.org/700812
12:36:46 gibi efried, dansmith: I finished reading the cyborg series. The re-schedule is not covered with functional test so I'm not sure it works. I also found other missing test coverage as well
12:38:15 rosmaita alex_xu: thank you
12:40:21 openstackgerrit Lee Yarwood proposed openstack/nova master: DNM test stable rescue tempest/devstack changes https://review.opendev.org/700815
12:41:07 sean-k-mooney gibi: looking at the fake cyborg driver i dont imediatly see where they are setting the RP name meaning they my be hardcoding it above the dirver meaning im not sure you coudl ever have more then one cyborg rp. ill keep looking but has anyone tested multinode with 2 hosts that have fpgas?
12:41:46 openstackgerrit Luyao Zhong proposed openstack/nova master: bug-fix: Reject live migration with vpmem https://review.opendev.org/708110
12:41:47 openstackgerrit Luyao Zhong proposed openstack/nova master: address specific resources cleanup issue https://review.opendev.org/699148
12:41:47 openstackgerrit Luyao Zhong proposed openstack/nova master: support live migration with vpmems https://review.opendev.org/687856
12:41:57 gibi sean-k-mooney: thanks for the answers in https://review.opendev.org/#/c/673735 much appreciated
12:42:22 gibi sean-k-mooney: I have to reload the context but months ago I was able to found the place where the hard-coding was done
12:42:44 sean-k-mooney the repo is small so it wont take me long to find it
12:44:27 sean-k-mooney i dont think i want to know why the cyborg agent calls ethe cyborg conductor to diff ther ereport data as part of updating the inventory do i...
12:44:52 sean-k-mooney agents shoudl really not do that
12:45:56 openstackgerrit Lee Yarwood proposed openstack/nova master: DNM test stable rescue tempest/devstack changes https://review.opendev.org/700815
12:48:35 sean-k-mooney gibi: ... so cyborg are doing all the inventory updates via the conductor by haveing the agents inventory the local host then doign an rpc to the conductor which does the diff of the old state to the new state then it updates the placement invtories
12:49:12 sean-k-mooney that is not goign to scale
12:50:57 sean-k-mooney also why are they using the dns_namespace for https://github.com/openstack/cyborg/blob/369abe8dd06aa6648298c3256f444a63ee6268d0/cyborg/conductor/manager.py#L417-L418
12:52:14 gibi sean-k-mooney: most probably they want to have stable uuids
12:52:36 gibi but I don't know why the DNS namespace is used for that
12:52:39 sean-k-mooney yes but they should have defiend there own namespace also the shoudl be using uuid5 not uuid3
12:52:58 gibi sean-k-mooney: agree
12:53:11 gibi uuid5 with a hardcoded namespace would be good
12:53:19 sean-k-mooney yep
12:53:31 sean-k-mooney md5 is old an busted and not fips compliant
12:54:08 sean-k-mooney uuid5 uses sha1 which is slightly less old, is fips complant and has hardwar instruction in the cpu
12:57:11 gibi honestly I lost in the code when I wanted to track down the RP naming
12:57:49 gibi the code change significantly since last time I looked into it
12:57:49 sean-k-mooney the painful thing is to change it you basicaly now need to do a cuckoo hash implemation to fall back to the old way of generating the uuid if you did not find it
12:58:36 sean-k-mooney gibi: ya i think the code will have to be refactored on there end before it can go into production as there is no way doing all the updated form the conductor like that will work for large deployments
12:59:25 sean-k-mooney the current architure will ddos the cyborg conductor with the periodic task updates
13:09:34 sean-k-mooney ah i found it
13:10:19 gibi sean-k-mooney: ohh, link it to me please :)
13:10:25 sean-k-mooney https://github.com/openstack/cyborg/blob/369abe8dd06aa6648298c3256f444a63ee6268d0/cyborg/accelerator/drivers/fake.py#L70
13:10:51 sean-k-mooney when they generate the deployable list https://github.com/openstack/cyborg/blob/369abe8dd06aa6648298c3256f444a63ee6268d0/cyborg/accelerator/drivers/fake.py#L59
13:10:54 gibi sean-k-mooney: thanks.
13:11:00 sean-k-mooney they use the device key form the dict
13:11:08 gibi yepp, I saw that
13:11:08 sean-k-mooney at least in the fake driver
13:11:28 gibi basically they need to include the host name when they convert the device name to RP name
13:11:36 sean-k-mooney and the conductor uses the name to get the uuid form the deployable
13:11:42 sean-k-mooney yep
13:11:49 sean-k-mooney im going to hack that in locally
13:13:38 sean-k-mooney basically driver_dep.name = "%s_%s" % (hostname,pci.get('device'))
13:13:53 gibi yep
13:14:12 sean-k-mooney although i might use the conf.host vlaue instead of the hostname
13:14:32 sean-k-mooney there is a utility function i think for this already
13:29:39 gmann thanks gibi
13:30:16 sean-k-mooney gibi: ok cool that appears to fix the placement issue but im still geting keystone auth issues
13:31:17 sean-k-mooney spcifically devstack@cyborg-api.service[8430]: 2020-02-18 13:28:51.849 8431 WARNING keystonemiddleware.auth_token [req-640319ce-6b48-48f5-a406-9038b51d0e67 cb8c1a4ddbc84fc5aadf961423c0a843 8ad0765aa5f84400b394ee05794f6402 - default default] Authorization failed for token: keystonemiddleware.auth_token._exceptions.InvalidToken: Token authorization failed
13:32:11 sean-k-mooney the token im sendign form the client is valid as i can do server create/delete so either the nova code is passing an expired token or more likely the cyborg api is not handeling it correctly
13:36:57 brinzhang_ gibi, sean-k-mooney: I have seen, thanks for update that, I think it's useful for some want to review some working patches.
13:38:25 sean-k-mooney ok so if i restart the nova service then the auth issue goes away i think but i think get other errors
13:38:28 brinzhang_ s/some/someone
13:38:47 sean-k-mooney on the plus side it did reschduler but i hav enot read through the logs to see if it did the right thing
13:43:23 sean-k-mooney gibi: dansmith efried those are the logs for the failed boot that tried to resudule. http://paste.openstack.org/show/789700/
13:43:43 sean-k-mooney i need to go offline for a few hours i will be back online this evening for an hour or so
13:46:36 sean-k-mooney the instance uuid is 4cefbecb-7ed1-4331-84d9-03f8d13f3ef6 but there looks like ther there is an error in how the compute mnager loops over the arqs and ther emight be other issues
13:55:38 openstackgerrit John Garbutt proposed openstack/nova master: WIP: Enforce unified limits using oslo.limit https://review.opendev.org/615180
13:58:11 openstackgerrit John Garbutt proposed openstack/nova master: DNM: Turn on unified limits to see if tests pass https://review.opendev.org/708387
14:22:47 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove 'nova.image.api' module https://review.opendev.org/702451
14:22:48 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove unnecessary exception handling, mocks https://review.opendev.org/703973
14:22:48 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove unused nova-network objects https://review.opendev.org/697156
14:22:49 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove unused parameters https://review.opendev.org/703974
14:30:54 dansmith sean-k-mooney: ah, nice, thanks
14:39:03 efried gibi: thanks for refreshing runways ++
14:53:58 dansmith sean-k-mooney: any chance you've got your rpc version pinned? as gibi noted in the review, that could cause that to be None there
15:12:27 dansmith sean-k-mooney: nevermind, I found the problem
15:29:56 gibi Sundar: It might be a sensitive question but how the re-org, that affects efried, effects your work in OpenStack?
15:30:43 gibi I'm asking this as I feel the cyborg integration work still need a sizeable effort to continue
15:31:23 dansmith gibi: I believe he'll be moving on at the end of march as well
15:31:45 gibi dansmith: then I'm affraid how will we finish the integration work
15:32:26 dansmith gibi: well, I expect it to get picked up by someone that is sticking around
15:32:51 gibi dansmith: yeah that would help
15:33:05 dansmith gibi: I've been avoiding actually working on the patches to retain my +2 legitimacy, but if that has to happen then I probably will, depending on the amount of work
15:34:04 dansmith gibi: I'm a bit confused about why you say the delete patch is where the reschedule work is going to happen because I don't see that patch even touching the requisite file(s). I'm guessing you're just okay with the earlier patch on the promise that reschedule will be accounted for somewhere later in the series?
15:34:30 gibi dansmith: thanks, that makes me less nervous
15:34:53 gibi dansmith: I followed Sundar pointer in his comment
15:35:12 gibi dansmith: but honeslty I need test coverage first to accept that it works
15:35:49 dansmith gibi: all he said is that a later patch *deletes* the original ARQs, but as far as I can tell, nowhere does it ever create them again on reschedule,
15:36:02 dansmith and the earlier create patch is kinda where I would expect that to happen
15:36:18 dansmith i.e. even if we orphan them until the delete patch or something
15:36:31 gibi dansmith: fair point, I will drop my +2
15:39:04 openstackgerrit Ivaylo Mitev proposed openstack/nova master: Faults from cell DB missing in GET /servers/detail https://review.opendev.org/699176
15:39:40 Sundar gibi, dansmith: I hope to finish the work. Brin Zhang is also reviewing and following this series to provide continuity. For the remaining work, I may need pointers for the rescheduling stuff. I thought, once we delete the ARQs, the conductor is going to pick another host from the allocation candidates and redo the previous flow.
15:40:09 openstackgerrit Ivaylo Mitev proposed openstack/nova master: Faults from cell DB missing in GET /servers/detail https://review.opendev.org/699176
15:40:22 dansmith Sundar: it does, but the way that happens is compute calls back to conductor through a different method, conductor pops an alternate host off the list and the calls to that compute to do the build again
15:40:32 dansmith Sundar: check my latest comments, I think I provided the necessary pointer
15:41:00 Sundar dansmith: Sure, going through all of your comments now
15:41:08 gibi Sundar: thanks. besides dansmith's pointer I strongly suggest to add functional tests as that helps you figuring out what is missing from the implementation
15:41:28 dansmith agree, obviously
15:43:03 dansmith sean-k-mooney: do you have a devstack localrc for cyborg testing with the fake driver that I can copy?

Earlier   Later