Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-02-18
12:53:31 sean-k-mooney md5 is old an busted and not fips compliant
12:54:08 sean-k-mooney uuid5 uses sha1 which is slightly less old, is fips complant and has hardwar instruction in the cpu
12:57:11 gibi honestly I lost in the code when I wanted to track down the RP naming
12:57:49 gibi the code change significantly since last time I looked into it
12:57:49 sean-k-mooney the painful thing is to change it you basicaly now need to do a cuckoo hash implemation to fall back to the old way of generating the uuid if you did not find it
12:58:36 sean-k-mooney gibi: ya i think the code will have to be refactored on there end before it can go into production as there is no way doing all the updated form the conductor like that will work for large deployments
12:59:25 sean-k-mooney the current architure will ddos the cyborg conductor with the periodic task updates
13:09:34 sean-k-mooney ah i found it
13:10:19 gibi sean-k-mooney: ohh, link it to me please :)
13:10:25 sean-k-mooney https://github.com/openstack/cyborg/blob/369abe8dd06aa6648298c3256f444a63ee6268d0/cyborg/accelerator/drivers/fake.py#L70
13:10:51 sean-k-mooney when they generate the deployable list https://github.com/openstack/cyborg/blob/369abe8dd06aa6648298c3256f444a63ee6268d0/cyborg/accelerator/drivers/fake.py#L59
13:10:54 gibi sean-k-mooney: thanks.
13:11:00 sean-k-mooney they use the device key form the dict
13:11:08 gibi yepp, I saw that
13:11:08 sean-k-mooney at least in the fake driver
13:11:28 gibi basically they need to include the host name when they convert the device name to RP name
13:11:36 sean-k-mooney and the conductor uses the name to get the uuid form the deployable
13:11:42 sean-k-mooney yep
13:11:49 sean-k-mooney im going to hack that in locally
13:13:38 sean-k-mooney basically driver_dep.name = "%s_%s" % (hostname,pci.get('device'))
13:13:53 gibi yep
13:14:12 sean-k-mooney although i might use the conf.host vlaue instead of the hostname
13:14:32 sean-k-mooney there is a utility function i think for this already
13:29:39 gmann thanks gibi
13:30:16 sean-k-mooney gibi: ok cool that appears to fix the placement issue but im still geting keystone auth issues
13:31:17 sean-k-mooney spcifically devstack@cyborg-api.service[8430]: 2020-02-18 13:28:51.849 8431 WARNING keystonemiddleware.auth_token [req-640319ce-6b48-48f5-a406-9038b51d0e67 cb8c1a4ddbc84fc5aadf961423c0a843 8ad0765aa5f84400b394ee05794f6402 - default default] Authorization failed for token: keystonemiddleware.auth_token._exceptions.InvalidToken: Token authorization failed
13:32:11 sean-k-mooney the token im sendign form the client is valid as i can do server create/delete so either the nova code is passing an expired token or more likely the cyborg api is not handeling it correctly
13:36:57 brinzhang_ gibi, sean-k-mooney: I have seen, thanks for update that, I think it's useful for some want to review some working patches.
13:38:25 sean-k-mooney ok so if i restart the nova service then the auth issue goes away i think but i think get other errors
13:38:28 brinzhang_ s/some/someone
13:38:47 sean-k-mooney on the plus side it did reschduler but i hav enot read through the logs to see if it did the right thing
13:43:23 sean-k-mooney gibi: dansmith efried those are the logs for the failed boot that tried to resudule. http://paste.openstack.org/show/789700/
13:43:43 sean-k-mooney i need to go offline for a few hours i will be back online this evening for an hour or so
13:46:36 sean-k-mooney the instance uuid is 4cefbecb-7ed1-4331-84d9-03f8d13f3ef6 but there looks like ther there is an error in how the compute mnager loops over the arqs and ther emight be other issues
13:55:38 openstackgerrit John Garbutt proposed openstack/nova master: WIP: Enforce unified limits using oslo.limit https://review.opendev.org/615180
13:58:11 openstackgerrit John Garbutt proposed openstack/nova master: DNM: Turn on unified limits to see if tests pass https://review.opendev.org/708387
14:22:47 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove 'nova.image.api' module https://review.opendev.org/702451
14:22:48 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove unnecessary exception handling, mocks https://review.opendev.org/703973
14:22:48 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove unused nova-network objects https://review.opendev.org/697156
14:22:49 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove unused parameters https://review.opendev.org/703974
14:30:54 dansmith sean-k-mooney: ah, nice, thanks
14:39:03 efried gibi: thanks for refreshing runways ++
14:53:58 dansmith sean-k-mooney: any chance you've got your rpc version pinned? as gibi noted in the review, that could cause that to be None there
15:12:27 dansmith sean-k-mooney: nevermind, I found the problem
15:29:56 gibi Sundar: It might be a sensitive question but how the re-org, that affects efried, effects your work in OpenStack?
15:30:43 gibi I'm asking this as I feel the cyborg integration work still need a sizeable effort to continue
15:31:23 dansmith gibi: I believe he'll be moving on at the end of march as well
15:31:45 gibi dansmith: then I'm affraid how will we finish the integration work
15:32:26 dansmith gibi: well, I expect it to get picked up by someone that is sticking around
15:32:51 gibi dansmith: yeah that would help
15:33:05 dansmith gibi: I've been avoiding actually working on the patches to retain my +2 legitimacy, but if that has to happen then I probably will, depending on the amount of work
15:34:04 dansmith gibi: I'm a bit confused about why you say the delete patch is where the reschedule work is going to happen because I don't see that patch even touching the requisite file(s). I'm guessing you're just okay with the earlier patch on the promise that reschedule will be accounted for somewhere later in the series?
15:34:30 gibi dansmith: thanks, that makes me less nervous
15:34:53 gibi dansmith: I followed Sundar pointer in his comment
15:35:12 gibi dansmith: but honeslty I need test coverage first to accept that it works
15:35:49 dansmith gibi: all he said is that a later patch *deletes* the original ARQs, but as far as I can tell, nowhere does it ever create them again on reschedule,
15:36:02 dansmith and the earlier create patch is kinda where I would expect that to happen
15:36:18 dansmith i.e. even if we orphan them until the delete patch or something
15:36:31 gibi dansmith: fair point, I will drop my +2
15:39:04 openstackgerrit Ivaylo Mitev proposed openstack/nova master: Faults from cell DB missing in GET /servers/detail https://review.opendev.org/699176
15:39:40 Sundar gibi, dansmith: I hope to finish the work. Brin Zhang is also reviewing and following this series to provide continuity. For the remaining work, I may need pointers for the rescheduling stuff. I thought, once we delete the ARQs, the conductor is going to pick another host from the allocation candidates and redo the previous flow.
15:40:09 openstackgerrit Ivaylo Mitev proposed openstack/nova master: Faults from cell DB missing in GET /servers/detail https://review.opendev.org/699176
15:40:22 dansmith Sundar: it does, but the way that happens is compute calls back to conductor through a different method, conductor pops an alternate host off the list and the calls to that compute to do the build again
15:40:32 dansmith Sundar: check my latest comments, I think I provided the necessary pointer
15:41:00 Sundar dansmith: Sure, going through all of your comments now
15:41:08 gibi Sundar: thanks. besides dansmith's pointer I strongly suggest to add functional tests as that helps you figuring out what is missing from the implementation
15:41:28 dansmith agree, obviously
15:43:03 dansmith sean-k-mooney: do you have a devstack localrc for cyborg testing with the fake driver that I can copy?
15:43:06 dansmith or Sundar ?
15:43:10 gibi also what sean-k-mooney's does while testing the feature is highly appreciated but those tests are not reproducible for us
15:43:38 gibi jingx, yeah at least a localrc would be good :)
15:44:33 dansmith gibi: test coverage in this set has always been majorly lacking, so given the time constraint, I was going to try to relax my expectations a bit, look to sean-k-mooney for validation of some things, and potentially follow-up with the functional testing after we've got things working
15:44:43 dansmith to keep my fingers out of it
15:44:55 dansmith not ideal, but the situation isn't ideal
15:45:16 gibi dansmith: understood
16:06:40 Sundar dansmith: Here's my local.conf -- http://paste.openstack.org/show/789708/
16:07:28 dansmith Sundar: thanks
16:09:14 dansmith Sundar: okay the only real change in there is enable_plugin cyborg
16:09:34 dansmith but if that's close enough, I can handle the rest of getting it to run on your patches I think
16:12:05 Sundar Yes, that's the main change. The default cyborg.conf has the fake driver enabled.
16:14:15 Sundar dansmith: We have to create device profiles and modify the flavor. Here are some utility functions to do that: http://paste.openstack.org/show/789709/
16:17:29 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Remove FakeScheduler https://review.opendev.org/707224
16:17:29 openstackgerrit Stephen Finucane proposed openstack/nova master: conf: Deprecate '[scheduler] driver' https://review.opendev.org/707225
16:17:30 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Improve documentation on writing custom scheduler filters https://review.opendev.org/707226
16:17:30 openstackgerrit Stephen Finucane proposed openstack/nova master: api: Add framework for extra spec validation https://review.opendev.org/704643
16:17:31 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Use recognized extra specs in tests https://review.opendev.org/708435
16:17:31 openstackgerrit Stephen Finucane proposed openstack/nova master: api: Add microversion 2.82, extra spec validation https://review.opendev.org/708436
16:19:03 stephenfin Not sure if CI will pass but if it does, I think that's basically done now ^
16:19:31 openstackgerrit Merged openstack/nova master: set default value to 0 instead of '' https://review.opendev.org/706730
16:33:11 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Use recognized extra specs in tests https://review.opendev.org/708435
16:33:12 openstackgerrit Stephen Finucane proposed openstack/nova master: api: Add framework for extra spec validation https://review.opendev.org/704643
16:33:12 openstackgerrit Stephen Finucane proposed openstack/nova master: api: Add microversion 2.82, extra spec validation https://review.opendev.org/708436
18:11:30 spatel sean-k-mooney: hey!
18:57:17 openstackgerrit Merged openstack/nova master: Add JSON schema and test for network_data.json https://review.opendev.org/703133
19:48:25 sean-k-mooney dansmith: gibi just getting back ill post my local.confs for both the all-in-one node and the dedicated compute latert today. ill try and adress the issue with the fake driver to allow multi node and fix the devstack plugin tonight/tomorow and see if i can create a multinode test jobs quickly with zuul over the next day or two.
19:49:31 dansmith sean-k-mooney: okay I got a single node devstack running with it, but got distracted.. I'll circle back tomorrow and look at whatever you have to offer and continue on
19:50:05 sean-k-mooney ya so the current fake driver does not work multi node because the RP name is hardcoded so its a tival change to make that work
19:50:39 sean-k-mooney and currently the cyborg config file is only compelete correctly if you deploy the conducto/api service so it does not work on the dedicated compute node
19:51:13 sean-k-mooney both are easy fixs once i have patch up for that it should not be hard to create a two node jobs and run standard tempest with a cyborg flavor
19:51:31 sean-k-mooney kind of like i did for the pcpu testing

Earlier   Later