Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-02-26
16:12:28 sean-k-mooney it should not fail in either case
16:12:47 sean-k-mooney once it start happening you have to restart all the nova services to fix it
16:22:50 sean-k-mooney efried: so no https://48ef08cde8cc22034a1d-8011a2266d21f0c09baf1c83d6d5002e.ssl.cf5.rackcdn.com/709641/5/check/cyborg-multinode-tempest-full/e4d260f/controller/logs/etc/nova/nova_conf.txt
16:23:12 sean-k-mooney the service user section https://docs.openstack.org/nova/latest/configuration/config.html#service-user
16:24:14 sean-k-mooney is not configured but it should not need to be confitured for it to work
16:24:44 sean-k-mooney it might allow you to mask the issue
16:27:35 efried sean-k-mooney: so the failure is happening from a later operation, not from the middle of a long-running operation?
16:46:05 gmann brinzhang__: sure, sorry for late response. I will review your patches today
16:50:40 efried sean-k-mooney: can I get your nod on https://review.opendev.org/#/c/709902/ (rocky EM patch) for os-vif please?
17:06:32 efried lyarwood: https://review.opendev.org/#/c/709902/1/deliverables/rocky/nova.yaml lgty?
17:12:08 lyarwood efried: ack yes, elod ^?
17:33:03 sean-k-mooney efried: it can start failing with a 401 in the middel of an operation and once it has it will continue to fail for seperate operations
17:33:41 sean-k-mooney and yes ill look at the em patch now
17:35:28 sean-k-mooney there is one pending bugfix i want to back port to all affected branch in os-vif but rocky predates the issue so yes that commit looks correct
17:35:32 sean-k-mooney ill +1 the review
17:38:23 sean-k-mooney efried: basicaly the way i first hit the cybrog issue was i booted a vm. did a bunch of life cycle operation on it and then tried to delete it and that failed befaue the token was rejected.
17:38:36 sean-k-mooney when i tried to do the operation (deleting the arq) myself it worked
17:39:25 sean-k-mooney if i listed device profiles myslef it also worked but when i tried to boot anotuher vm after that point it failed beacuse the nova api was not able to retive the device profile info
17:40:44 sean-k-mooney so if i use osc to query cyborg directly everthing is fine. if i use osc to boot a vm and nova tires to query cyborg on my behalf it was failing but only after the services had been running for a while like an hour or so
18:24:17 lucidguy Anyone recall me asking for assistance with >1tb ram instances? I FIGURED IT OUT!!! only took days.
18:25:23 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Provide the backing file format when creating qcow2 disks https://review.opendev.org/708745
18:27:10 lyarwood lucidguy: what was it?
18:27:11 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Provide the backing file format when creating qcow2 disks https://review.opendev.org/708745
18:32:44 sean-k-mooney updating other people code when you have never reviewed it and are just following gerrit comments is hard
18:33:19 sean-k-mooney the changes arent hard but the mental load to make sure what you are doing is correct is way higher then when its your code
18:39:37 melwitt lucidguy: don't leave us hanging
18:44:30 sean-k-mooney lucidguy: was it the alingment of jupiture
18:44:50 sean-k-mooney *jupiter
19:15:22 openstackgerrit Merged openstack/python-novaclient master: Don't print user_data for 'nova show' https://review.opendev.org/708850
19:51:16 lucidguy sean-k-mooney?
19:51:19 openstackgerrit Lee Yarwood proposed openstack/nova master: DNM - Test stable device rescue tests with BFV instances https://review.opendev.org/710050
19:53:41 lucidguy sean-k-mooney: By default instances are launched with 40bit cpu memory address space, that does not allow for >1tb memory in an instances. qemu on Ubuntu 18.04 allow to choose a machine architecture that maps the instances address space with the local HV which is 46bits. In the end of the day all I had to do is upgrade to 18.04(Bionic) and add one line to nova.conf on the HV.
20:08:10 openstackgerrit sean mooney proposed openstack/nova master: Provider Config File: YAML file loading and schema validation https://review.opendev.org/673341
20:20:20 sean-k-mooney lucidguy: ah yes that makes sense
20:20:40 sean-k-mooney intel cpus only recently went to 48bit adress space
20:21:07 sean-k-mooney so im nost surpiesed the same limiation of reduced adress space was present for vms
20:21:48 sean-k-mooney lucidguy: did you fix it by changing the machine type to q35?
20:22:02 sean-k-mooney lucidguy: or did you add something else to the nova.conf
20:22:11 sean-k-mooney for example the cpu_model?
20:23:25 sean-k-mooney also a stackdump is not an approriate way of telling the enduser that you need to change somthing like that
20:23:46 sean-k-mooney i hope they have adressed that in a future version of qemu/kvm
20:24:40 sean-k-mooney gibi: efried: i tried to keep my changes in https://review.opendev.org/#/c/673341/ as minimal as possible while adressing the comments.
20:25:32 sean-k-mooney i will try to get through the other patches in the series tommorow. there were some race condition in the test code that took me a while to figure out
20:25:48 sean-k-mooney they are fixed now
23:11:23 efried sean-k-mooney: Earlier, we were toying with cutting cyborg over to using sdk instead of ksa. We held off because there were a couple more quirks to be ironed out. It's possible sdk would automatically refresh the token for us -- mordred?
23:15:18 efried sean-k-mooney: It's been a minute, but I think on the code side you just have to s/get_ksa_adapter/get_sdk_adapter/ to make the switch. If so, perhaps we could stuff that change in between the series and your tester somehow and see if it fixes the problem.
23:16:11 efried If so, then unit/functional tests would just need small tweaks to make it go.
23:36:57 mordred efried: sdk in general should refresh tokens
23:37:11 mordred efried: however, it's possible there are specifics I should page in
23:37:18 mordred also - we just landed cyborg support in sdk
23:38:03 mordred efried: I'm EOD today - but I'd be happy to interact with folks on it tomorrow to see if we need to do anything
#openstack-nova - 2020-02-27
01:07:54 openstackgerrit Brin Zhang proposed openstack/nova master: Add test coverage of existing os-volumes-attachments policies https://review.opendev.org/709929
01:31:59 openstackgerrit Brin Zhang proposed openstack/nova master: Introduce scope_types in os-volumes-attachments policy https://review.opendev.org/709388
02:26:50 rm_work gibi: reviewed the routed-network tempest test -- that test needs a bit of cleanup
06:52:23 openstackgerrit Brin Zhang proposed openstack/nova master: Add new default roles in os-volumes-attachments policies https://review.opendev.org/710190
08:08:15 aarents Hi there, gibi when you will have time for that, can you confirm that https://review.opendev.org/#/c/702368/ is ok for you, I fixed the nits you reported.
08:22:15 gibi rm_work: thank you for the tempest review. I've started working on the pre-filter patch from mriedem
08:22:26 gibi aarents: opening the patch now...
08:22:28 rm_work :thumbsup:
08:31:27 gibi stephenfin: a relatively easy bugfix needs a second code https://review.opendev.org/#/c/702368
08:31:39 gibi aarents: thanks for the bufix, I'm +2 on it
08:32:03 gibi stephenfin: s/code/core/ // I need more coffeee
08:32:30 aarents gibi: thks
08:48:54 lyarwood kashyap / stephenfin ; https://review.opendev.org/#/c/708745/ - a simple partial fix for the libvirt v6.0.0 qcow2 backing file format issue if you have time to review today.
08:54:58 kashyap lyarwood: Morning, will look
08:56:21 kashyap lyarwood: The discussions upstream turned even more complex; but I think we should go with this fix for now
08:56:29 kashyap Looks good to me
10:30:19 openstackgerrit Johannes Kulik proposed openstack/nova master: Validate id as integer for os-aggregates https://review.opendev.org/710214
10:34:37 jkulik regarding ^ afaics, osc-lib first tries to get() whatever is supplied to `openstack aggregate show`, which the os-aggregates endpoint doesn't handle well, if it's a name and not an id.
10:34:53 jkulik resulted in too many 500 errors in our environment
10:51:38 kashyap lyarwood: Reviewed; I also posted the upstream libvirt conclusion for selective format probing (and not letting it recurse into multiple levels of backing chains)
10:51:51 kashyap (Read this one: https://www.redhat.com/archives/libvir-list/2020-February/msg01001.html)
10:58:17 lyarwood kashyap: ack thanks
11:45:30 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Use 'import foo from bar' https://review.opendev.org/710235
12:01:27 openstackgerrit Stephen Finucane proposed openstack/nova master: virt: Remove QEMU_VERSION_REQ_SHARED https://review.opendev.org/710239
12:01:27 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_MULTIATTACH https://review.opendev.org/710238
14:10:40 openstackgerrit Stephen Finucane proposed openstack/nova stable/train: Recalculate 'RequestSpec.numa_topology' on resize https://review.opendev.org/708340
14:11:13 stephenfin lyarwood: I rebased that onto one of your patches to avoid a merge conflict. Any change you could re +2? ^
14:14:13 lyarwood stephenfin: done, elod ^ if you have a chance btw.
14:25:51 openstackgerrit Stephen Finucane proposed openstack/os-traits master: setup.cfg: Configure 'python-requires' https://review.opendev.org/704534
14:25:52 openstackgerrit Stephen Finucane proposed openstack/os-traits master: Switch to hacking 2.x https://review.opendev.org/710275
14:25:52 openstackgerrit Stephen Finucane proposed openstack/os-traits master: Remove use of six https://review.opendev.org/710274
14:25:53 openstackgerrit Stephen Finucane proposed openstack/os-resource-classes master: Switch to hacking 2.x https://review.opendev.org/710276
14:26:25 stephenfin bauzas, gibi, efried: Some dead simple python27 cleanup patches there, if you want the stats ^
14:26:50 bauzas heh
14:29:41 efried sean-k-mooney: did you see my comments yesterday about ksa->sdk conversion?
14:33:58 lyarwood https://review.opendev.org/#/c/708745/ - For any non-RH cores, this is a simple partial fix for an issue introduced recently in Libvirt v6.0.0.
14:34:17 sean-k-mooney efried: no
14:34:43 sean-k-mooney i was logged off ill look at irc logs unless you want to summerise
14:34:54 efried sean-k-mooney: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2020-02-26.log.html#t2020-02-26T23:11:23
14:35:00 efried not too much to summarize
14:35:59 sean-k-mooney ah i see
14:36:21 efried I don't like the idea of un-singleton-ing, which would mean we do the whole Adapter setup with every API call, versus putting in something that will keep the token alive for us. If that thing is converting to sdk, win/win.
14:36:28 sean-k-mooney so ya my theory is that by making it a singolton in the current patch we are not refrshing the token
14:36:45 sean-k-mooney so the fact we were always createing the adapters in other case was masking that
14:36:53 efried yup
14:37:28 sean-k-mooney so unless we do the swich i think the singolton is wrong or at least we would need to add an expiry/token check to it if we kept it
14:38:18 sean-k-mooney i have been using https://review.opendev.org/#/c/679914/
14:38:21 sean-k-mooney in my testing
14:38:30 sean-k-mooney oh which is now merged cool

Earlier   Later